Yes, with conditions: commission an authorized penetration test before exposure when the app will handle sensitive data, privileged actions, or money. Otherwise, you usually need one only when a customer contract, security questionnaire, or auditor asks for it in writing. The exact wording tells you what evidence will be accepted.
AI pentesting actively probes a scoped, running target for security weaknesses and may attempt to exploit them. A dependency scan, source-code review, or launch-readiness audit can inform that work, but it does not become a penetration test simply because AI or security tooling is involved.
That distinction matters here. AxonBuild repairs agreed problems in working AI-built apps. For a new client, one qualifying blocker may be a $99 first job, with payment after you see it work. We inspect the relevant code after agreeing to take the blocker. This is not penetration testing, exploit coverage, or regulated certification. If you need evidence that a weakness can be exploited against a live target, commission an authorized pentest with that objective clearly authorized.
The useful question is therefore not whether an AI pentest is “verified.” It is what evidence supports each finding, what environment was tested, which actions were allowed, and what the engagement did not test.
Is an AI pentest enough for SOC 2 or a customer requirement?
Sometimes, and only the requester can tell you. Whether penetration testing is required for SOC 2 depends on the control your auditor is testing and on the contract clause behind the request. Read the exact wording first. An AI-run test either satisfies it or it does not.
Only the exact control, contract, assessor, or customer can answer that. Some vendors market auditor-ready reports, but acceptance depends on the specific test and on who is reading the report. Prescriptive standards can require defined methodology, work boundaries, tester qualifications or independence, frequency, and retesting. AxonBuild’s $99 first job and larger repair work cannot satisfy a request for a penetration test, because neither one is a penetration test.
So the question to answer is not “is penetration testing required for SOC 2” in general. It is which control, contract clause, or customer question you are trying to close, and what that specific wording will accept as evidence. Get that in front of you before you price anything.
A customer or auditor asked for a penetration test. What do you actually need?
Get the request in writing, then match it to a test type. These requests usually come from an enterprise buyer’s security review or a SOC 2 auditor, and both normally want an independent test of the running app with a report you can share.
Four wordings show up repeatedly, and they do not ask for the same thing:
- “Provide a recent penetration test report.” A dated report against the production or staging application, usually within the last year. The requester cares about the date, the target, and who performed it.
- “Confirm you perform annual penetration testing.” A recurring commitment, not one report. A questionnaire answer plus evidence of the last run is normally what lands here.
- “Share your vulnerability scan results.” A scan, not a pentest. Cheaper and faster, and a pentest report will usually cover it while the reverse is not true.
- “Our auditor needs evidence for this control.” Ask which control, then read its wording. What satisfies it may be a scan, a test, a written policy, or a ticket showing a finding was fixed.
Penetration testing for SaaS carries one detail that catches small teams out: the app runs on someone else’s infrastructure, so their testing rules apply too. AWS, for example, permits customer penetration testing against a published list of services without prior approval, requires approval for command-and-control testing, and prohibits denial-of-service and simulated denial-of-service testing under that policy, with DDoS simulation handled separately through AWS’s own DDoS simulation testing policy. Read your provider’s policy before anyone starts, not after.
Choosing between a scan, a code review and a pentest is a separate decision, and comparing the AI security scanners themselves is a separate job again.
If the request is not written down, ask for it in writing before choosing a test. That step defines the evidence the requester will accept. It does not replace the separate risk trigger for sensitive data, privileged actions, or money, which applies before exposure.
What counts as AI penetration testing?
NIST defines penetration testing as security testing in which evaluators attempt to circumvent or defeat security features under specified constraints. AI can automate parts of reconnaissance, request generation, exploit selection, state tracking, evidence capture, and report drafting. The testing still needs a target, authorization, scope, and rules of engagement.
Several activities often appear under the same search term:
| Activity | Typical evidence | What it does not establish by itself |
|---|---|---|
| Dependency or configuration scan | A package, version, secret pattern, or unsafe setting is present | That an attacker can reach or exploit it |
| Static source review | A risky path exists in code and may be reachable from an entry point | That the deployed target behaves the same way |
| Dynamic security scan | The running application returned a suspicious response | That the behavior creates material impact |
| Exploit-validating pentest | Reproduction steps, requests, responses, and impact evidence from the scoped target | That every weakness in the target was found |
| Business-priority review | Consequence, owner, urgency, and remediation context | Live exploitability unless it was also tested |
Modern AI pentest products can reach the fourth row for some findings. Some can test broken authorization and business logic, produce working proofs of concept, and retest fixes. Their capability depends on access, test identities, target stability, allowed impact, product configuration, and the application itself. “AI pentest” is not one consistent methodology.
The products themselves sort into rough categories, and lumping them together hides more than it explains:
| Category | Representative tools | What it aims to prove |
|---|---|---|
| Autonomous web-app pentest | XBOW, Aikido Attack | A real exploit chain against a live app, with proof of exploitation |
| Exposure / reachability validation | Vendor scanners with attack-path checks | Whether a flagged CVE is actually callable, not just present in a dependency tree |
| Agentic / open framework | Cybersecurity AI (CAI) | Whatever you configure it to prove; requires real setup time |
| CI/CD runtime scanning | Pipeline-integrated SAST/DAST | Catches a new hole at ship time; blind to anything CI doesn’t exercise |
| AI-app red-teaming | Prompt-injection and jailbreak probes | Whether a user can talk your own model into something it shouldn’t do |
| Open-source point tools | BugTrace-AI, Shannon | Cheap, narrow, hands-on; strong on discovery, weaker on judgment |
Use an evidence ladder, not a verified badge
Verification has levels. A package advisory can be correct about an installed version while remaining irrelevant to the deployed app. A runtime response can reproduce a bug without proving meaningful impact. An exploit can demonstrate impact within one role while leaving other roles and workflows untested.
| Evidence level | A defensible report should show |
|---|---|
| Candidate | The rule, package, request, or code location that triggered the finding |
| Static reachability | The source entry point and call path, plus assumptions that were not tested at runtime |
| Runtime reproduction | Target, identity, preconditions, request, response, and repeatable result |
| Exploit impact | The authorized action achieved and evidence of its consequence, with sensitive artifacts protected |
| Business priority | Affected users or assets, likelihood under real controls, remediation owner, and accepted residual risk |
For black-box testing, file-and-line evidence may be impossible because the tester has no source access. For white-box testing, it is useful but still does not prove the deployed behavior. A good report labels the evidence level rather than using one word for all of them.
The AxonBuild audit corpus contains a useful example of why those levels should not be mixed. In three of 21 third-party app reviews, software-composition scans produced 33 to 44 dependency advisories and source tracing found no route from the app into the affected functions. The clearest of the three was a freelancer dashboard, a Bolt.new export with no backend, no database, no auth, and no payments; every invoice and client record on the page came from a hardcoded array. A dependency scan against it came back with 44 advisories. I traced every one. Almost all of them turned out to be build-tooling and dev-dependency noise that never ships to a browser, plus a single open-redirect advisory in the router with no user-controlled redirect for anyone to actually hijack. Zero reachable, and I could write that down with confidence because I’d checked each one, not because the report happened to be short. That app scored 81, the best in the corpus, largely because the audit could rule things out instead of just listing them.
That is evidence about dependency-advisory triage in those apps. It is not a false-positive benchmark for autonomous AI pentesting, and it does not show how an active tool would have behaved against the targets.
The published evidence is promising and limited
One unusually detailed comparison is Doyensec’s published evaluation of Aikido and XBOW. It reported 49 accepted true-positive findings and two false positives for Aikido, compared with 31 accepted true positives and one false positive for XBOW. Those counts correspond to false-positive ratios of roughly 4% and 3% within the study.
The limitations belong beside the numbers. Aikido sponsored the engagement and influenced its constraints, while Doyensec says it independently executed the tests and collected and presented the results. The evaluation used two applications, one configuration, and deliberately omitted false-negative measurement. Its authors state that the results are not generalizable. It demonstrates that both systems found and validated real issues under those conditions; it cannot establish either product’s overall coverage.
Open-source results need similar labels. A February 2026 Help Net Security practitioner report described one run of Shannon at about $8 to $10 in model usage and reported some unproductive BugTrace-AI leads. That is a useful experience report, not a benchmark or a durable price. Shannon’s current repository describes a different and evolving scope, warns that tests can mutate state, requires human review, and gives a higher example model-cost estimate. Target size, model, version, test depth, and time all change the result.
When comparing tools, prefer evidence from your own representative staging target. Published comparisons can help design that evaluation, but they cannot substitute for it.
Set the safety boundary before running a tool
Active testing can create accounts, change or delete data, send emails, exhaust resources, trigger webhooks, and reach third-party systems. Only test systems you own or have explicit written authorization to test.
Run autonomous security testing in staging or a disposable environment with synthetic data, a defined scope, and a tested stop path.
Before a run, document:
- The domains, APIs, roles, workflows, and third-party integrations that are in and out of scope.
- Written authorization, the testing window, rate limits, and prohibited actions.
- Dedicated test identities and synthetic data for each relevant permission level.
- Whether account creation, state mutation, file upload, email, payment, or destructive impact is allowed.
- A current recovery point, monitoring, a kill switch, and the person authorized to stop the run.
- How reports, credentials, session tokens, and exploit evidence will be stored and destroyed.
OWASP’s Web Security Testing Guide provides a broader framework for planning and reporting web application tests. A product’s autonomous mode does not remove the need for those engagement controls.
When to use a pentest, a code review, or both
Use an authorized pentest when the decision depends on deployed exploitability. Common examples include a customer contract requiring a pentest, penetration testing for SOC 2 evidence where the assessor has said a test is what they will accept, a payment or sensitive-data workflow that needs adversarial runtime testing, or a material release whose exposed attack surface has changed.
Use source and workflow review when the decision depends on design, ownership boundaries, incomplete implementation, deployment assumptions, maintainability, or prioritizing what to fix before an active test. This can identify likely authorization and business-logic paths, but the report must preserve any runtime validation gaps.
Commission an authorized penetration test before exposure when the app will handle sensitive data, privileged actions, or money. Source and workflow review can expose intended behavior and high-risk paths before the test challenges them in the deployed environment. Findings from either method still need remediation, retesting, and an explicit owner.
For an AI-built app, this sits alongside the broader launch-readiness decision and the recurring security failure patterns in AI-generated code. Neither article is a substitute for the test your actual risk or customer requirement calls for.
How to compare what pentest vendors offer
Price is meaningful only when the application, endpoints, roles, testing depth, and required evidence are comparable. Open-source software may carry model, setup, monitoring, and triage costs. Managed products commonly price by application size, endpoints, roles, testing depth, or usage, and current quotes change with the application size and the endpoints, roles, depth, or usage included. Traditional human-led engagements vary for the same reason.
Ask each provider the same questions:
- Is the work static, dynamic, exploit-validating, or a combination?
- Which authenticated roles and business workflows will be tested?
- What impact is the tester allowed to demonstrate?
- Does every reported issue include reproducible evidence, and how are informational candidates labeled?
- How are false negatives assessed, if at all?
- Is remediation guidance, a retest, and a final report included?
- Which contractual or compliance requirement is the report intended to satisfy?
A cheap run that covers one unauthenticated route is not comparable to a multi-role test with retesting. A larger finding count is not automatically better either. Compare coverage, evidence, limitations, and the decision the report needs to support.
Common questions about AI pentesting
What is AI pentesting?
AI pentesting is penetration testing in which a model or agent automates parts of the work: reconnaissance, request generation, exploit selection, evidence capture, and report drafting. The authorization, the target, and the rules of engagement are unchanged.
Can AI replace a human pentester?
AI systems can already automate meaningful parts of active testing and can produce exploit evidence for some vulnerabilities. Human responsibility remains important for authorization, scoping, safety, ambiguous business intent, evidence review, risk acceptance, and accountability. The right staffing model depends on the target and the consequence of being wrong.
How much does an AI pentest cost?
There is no reliable category-wide price. One Shannon run in the February 2026 Help Net Security report above cost about $8 to $10 in model usage, plus operator time. XBOW’s seller announcement, Announcing XBOW Pentest On-Demand for Security at Machine Speed, published November 13, 2025 at xbow.com/news/announcing-xbow-pentest-on-demand-for-security-at-machine-speed and checked August 17, 2026, priced its offering around $4,000. It compared that price with human-led tests at $10,000 to $35,000 and 35 to 100 days. Those latter figures are XBOW’s sales comparison, not market-wide prices or schedules. Endpoints, roles, depth, and retesting move the price. Request the targets, roles, workflows, allowed actions, retesting, and final report in writing, then compare the evidence delivered, not only the headline price.
What does a verified finding mean?
The report should define it. Look for the tested target and identity, preconditions, repeatable steps, observed result, authorized impact evidence, and limitations. Static reachability and runtime exploitation are both useful evidence, but they answer different questions.
What do AI pentest tools miss?
Anything outside the supplied scope, identities, accessible workflows, tool capabilities, or allowed impact may be missed. A tool can also find one path while missing a variation of it. Ask how coverage and false negatives were evaluated, then retest high-consequence fixes and assumptions independently.
Need this fixed in your own app?
New clients can start once with one agreed blocker for $99. We fix it within three business days once access works, and you pay after seeing it work.
