The test output says every check passed. A customer says the payment page spins for a while and then does nothing. Both of those statements can be true at the same time, and on an app built with an AI assistant they usually are.
This page is about that gap and only that gap. It does not teach you how to build a test suite and it does not rank testing tools. The suite already exists, an assistant wrote most of it, every check is green, and the app is broken anyway. The answer starts with what the green actually covers.
A passing test suite proves that the checks somebody wrote still hold. It proves nothing about the checks nobody wrote. Across 26 AI-built apps audited in June and July 2026, exactly one had a working suite, and that app still came out red.
Method, briefly. The corpus numbers here come from AxonBuild’s audits of 26 AI-built apps in June and July 2026, a fixed cohort that does not grow and never gets rounded up. The mechanism claims about AI-written checks are read from named, dated, published sources on the dates given. Nothing on this page is a hands-on run of a testing tool.
What a green test run actually proves
A test is a question somebody decided to ask, written down in a form a machine can re-ask on demand. “When a logged-in customer clicks pay with a valid card, does an order row appear?” is a question. Somebody had to think of it, decide it mattered, and write it down. Once it exists, the machine will ask it forever, on every push, for free.
That is genuinely useful, and it is also the whole of what a green run tells you: the questions on the list still get the same answers they got when the list was written. The run is completely silent on every question that never made the list. It does not know they exist. Nothing in the output distinguishes “somebody asked and it was fine” from “nobody ever asked”.
So a suite of 235 questions with a green light beside each one is a report about 235 questions. The customer who cannot pay you is standing on question 236.
Mike Kamaev opened a LinkedIn post on 4 August 2026 with the same arithmetic: “235 tests passed. The product was still broken.” The number is doing no work in that sentence. Two hundred and thirty-five is a count of decisions somebody made about what to check, not a measurement of the product.
Why the same thing wrote the code and the checks
On an app assembled by prompting, one more failure sits on top of the ordinary blind spots, and it is specific enough to name.
When you ask an assistant to add tests, it reads the code that exists and writes checks that match it. If the code is correct, you get a check that locks in correct behavior. If the code is wrong, you get a check that locks in the wrong behavior, and it passes, and it will keep passing every day for the rest of the app’s life.
David Adamo Jr. published a worked example of exactly this on 7 October 2025. A division function is supposed to raise an error when you divide by zero. The version in his example returns zero instead, which is wrong. The generated test asserts that dividing ten by zero returns zero, so the run is green and the bug is now protected. His summary of the mechanism is that “your tests now validate the implementation, not the intention”. He also makes the wider point that generated checks tend to replay what the code already does rather than enforce a rule anybody agreed to.
Kenneth Sanchez V, writing on dev.to in a piece called “Your AI Agent Says All Tests Pass. Your App Is Still Broken”, names the session-level version: “When the same agent writes the implementation and the tests in the same session, the tests become a mirror of the agent’s understanding, not a mirror of your intent.” A company called Autonoma has a post on the same mechanism at getautonoma.com/blog/ai-generated-tests-pass-but-dont-assert, framed around generated checks that run without asserting anything meaningful.
Translated out of developer language, that is the part an owner needs. The suite carries the same opinion as the code, written down twice, in two files, by the same author. When you go looking for confirmation that the app works, you are asking the builder whether the builder did a good job, and getting the answer in a font that looks like evidence. This holds whichever tool assembled the app, Lovable or Base44 or Bolt or Replit or Cursor or Claude Code, because the mechanism is the generation session rather than the vendor.
On 16 August 2026, every on-topic page ranking for this question was addressed to somebody who can open a test file and change it: Stack Overflow threads about order-dependent runs, a TypeScript test rig on dev.to, a QA vendor’s pitch. The one page written in an owner’s words was a thread in the r/vibecoding subreddit titled “All my tests passed, the app was completely broken”, at reddit.com/r/vibecoding/comments/1ui4nwm/all_my_tests_passed_the_app_was_completely_broken/.
Seven things a passing test suite cannot see
Here is the map, with what AxonBuild’s audits of those 26 apps found beside each item. Denominators differ by row on purpose: 21 of the apps were third-party and 5 were the founder’s own, and the corpus does not average across that line.
| What stays invisible | Why the run still goes green | What the audits found | What you can check yourself |
|---|---|---|---|
| Nothing at all covers the path that broke | The suite re-asks only the questions somebody wrote down, and that path was never one of them | At least 18 of 21 third-party apps had no working test anywhere (June and July 2026 cohort) | Ask which check covers the exact screen the customer was on when it failed |
| The checks never reach the real code | A test can call a stand-in and pass while the production path is never executed once | One retail point-of-sale app of those 21 ran a checkout suite that never called the code that creates a sale | Ask whoever built it to show you the check that runs the real payment path |
| The flows a stranger hits first are skipped | Sign-up and login are awkward to automate, so they get left out and the total still looks full | The single app of 26 credited with a real suite skipped sign-up, login, and the payment webhook | Make a brand new account in a private window and go all the way through to paying |
| The check asserts what the code does, not what you needed it to do | A generated check reads the implementation and matches it, so a wrong answer gets recorded as the expected answer | No corpus count applies; this one is a mechanism, documented by David Adamo Jr. in October 2025 and Kenneth Sanchez V on dev.to | Read one check’s name aloud as a sentence and ask whether that is a promise you care about |
| Nothing runs the suite before the code ships | The suite went green on somebody’s machine once and never ran again; some apps switch off their own type and lint checks at build time | At least 17 of 21 third-party apps had no deploy gate (June and July 2026 cohort) | Push an obvious mistake to a branch and see whether anything refuses it |
| The failure happens in production and nothing records it | The error reaches the customer and stops there, so nobody ever learns there was a question worth asking | 17 of 21 third-party apps recorded errors nowhere (June and July 2026 cohort) | Force a harmless failure on the live site, then look for it wherever you would expect an alert |
| The break is what the person sees | Assertions read the data behind the page rather than the page, so an unreachable or invisible control still passes | No corpus count applies; this one is a mechanism, documented by Ajay Kulkarni in April 2026 | Open the app on a phone you did not build it on and do the one thing customers pay for |
Row six has a sibling worth knowing about, because a suite reporting success it did not earn is one half of a pair. The other half is an app that swallows the error and still reports success to the browser while the work never happened. Both are the same lie told by a different part of the system: there the running app is the one saying everything is fine, here it is the suite.
Row seven is the one owners hit most often and recognize least, because the thing they are looking at is a screen and the thing the check is looking at is a database row. Ajay Kulkarni put it in four words in a LinkedIn article in April 2026: “Your tests don’t have eyes.” A button pushed off the visible area, a modal that opens behind the header, text the same color as its background: every one of those is a working app by assertion and a dead app by customer. Environment belongs in this row too. If the suite runs on a machine with different settings, different keys, or a different database than the live site, it can pass honestly and still say nothing about production, which is the case where the environment is what broke, not the code.
One word to retire while you are here. Developers call a check that passes sometimes and fails sometimes a flaky test, and most of the SERP for this question is about that. A flaky test is a check whose answer wobbles. Everything on this page is the opposite problem: a check whose answer is perfectly stable and about the wrong thing.
In the third-party half of the cohort, all 21 of them, Reliability and Correctness was the worst of the twelve scored pillars at 31.4 out of 100, while Secrets and Credentials was the best at 84.4. The stereotype about AI-built apps is leaked keys. What 26 audited AI-built apps actually failed at is the invisible engineering: nothing proving the app works, nothing watching it, nothing stopping a bad change from shipping.
The one audited app that had a real test suite
Out of all 26 apps in that cohort, exactly one was credited with a working automated test suite. It was a healthcare community hub built on Next.js, the kind of app clinicians and builders post to and read. Everyone else in the corpus had either nothing or something that did not run.
That app scored 48 out of 100 and came out red. To be precise about which one, because a second app in the same corpus also scored 48: this is the one app in the corpus with a working test suite, and the other 48 belongs to a different app entirely.
Two confirmed criticals put it there. The first was its flagship feature, an AI generator endpoint that answered a request from anybody with no login and no rate limit, and that quietly ran the model on the server’s own provider key when the caller did not supply one. A loop against that URL turns the owner’s model bill into an unbounded charge from strangers. The app’s two other AI endpoints throttled anonymous callers specifically to prevent this, and the rate-limit table even had empty buckets waiting. That one route was never wired to it.
The second was the framework itself. The pinned version carried published high-severity advisories, reachable on the live internet-facing app, including a denial-of-service path a stranger could trigger against the running pages. The fix was a version bump.
The suite was real, and it had never been pointed at either of the two things that made the app red.
No test failed on the way to those two findings, because no test was ever written that would have asked. The suite covered what somebody thought to cover, and it skipped sign-up, login, and the payment webhook, which is to say it skipped every door a stranger walks through first. A green run on that app was accurate and useless in the same instant.
This is the honest ceiling on what a good suite buys you. It protects the behavior somebody described, and leaves the behavior nobody described exactly where it was. The undescribed part is disproportionately the part that costs money, because money moves through the paths a builder assumes rather than the paths a builder writes down.
What to check by hand when the suite says green
Soobin Park, who built the app in question solo, described the discipline this asks for in a LinkedIn post on 16 April 2026: “I built a full-stack AI app solo, achieved a 100% E2E test pass rate, and then… I completely paused the rollout.” E2E, or end to end, means a check that drives the whole app the way a person would rather than one piece of it in isolation. A 100% test pass rate on that kind of suite is the strongest green an app can produce, and she still stopped.
You can do a smaller version of the same thing in an afternoon, with no ability to read code. Five checks, in this order:
- 01 Walk the exact path the customer walked, on the real deployment, not a preview link and not a local copy. Same browser, same account type, same starting page.
- 02 Make a second account and try to reach the first account's data. Change the number in the address bar, open a shared link while signed in as the wrong person, and see what comes back.
- 03 Try a paid action from an account that has never paid. If it works, the money path is decorative.
- 04 Break something on purpose, somewhere harmless, and see whether anyone is told. If no message arrives anywhere a human looks, production failures are already invisible to you.
- 05 Confirm that whatever deploys the app runs the suite first, and refuses to ship when it goes red. Ask for the setting, not the reassurance.
Check two catches what an author writing checks about their own code rarely covers, because the author was picturing the customer who is signed in correctly. Check five is the one people skip and then repeat forever: a suite that never runs before a deploy has retired.
Everything above tells you whether the app is broken right now, on the paths you happened to think of. It is a spot check, and it will not tell you whether the parts you did not walk are fine.
Does a 100% pass rate or 100% coverage mean anything?
A 100% pass rate and 100% coverage are both measurements of the test suite, and neither measures the product. The pass rate says every question on the list got its expected answer. Coverage says what share of the code ran while those questions were asked, which a check that asserts nothing still satisfies.
Owners read coverage as a completeness score. What it counts is visits: lines the test process passed through, with no opinion on decisions anybody checked, rules anybody enforced, or paths anybody thought about. That is how a suite reaches a high coverage number over a checkout while never executing the function that takes the money, which is what one audited retail point-of-sale app did.
A pass rate has the same problem one level up. It is the ratio of green checks to total checks, and the denominator is a list of decisions somebody made. Adding twenty easy checks to a suite of eighty moves the number and moves nothing about the app. Removing the two hardest checks makes the rate go up.
There is a real developer argument about what coverage thresholds are worth, and this is not it. For an owner, the useful reading is narrow: both numbers describe how thoroughly the suite examines itself. Whether the app works gets answered somewhere else, by walking the app or by having a person read the code.
When the answer is to have somebody read the code
More checking is the right instinct most of the time. Three situations are where it stops being the right next move.
The first is when money is already moving. Once real cards are being charged, the gap between “the suite covers it” and “a stranger cannot abuse it” is a business problem, and writing more checks against code you cannot read does not close it. The second is when a customer has already been hurt, because at that point you need to know how far the damage reached, and no test suite is going to tell you what a person did on your live app last Tuesday. The third is when nobody can say what the app is supposed to do in the first place. A check is a written-down decision, and if no decisions were ever made, generating more checks just produces more mirrors.
All three need a judgment call, and a suite cannot make one. The next step in each case is paying somebody to read the code, which produces something a run cannot: a person looking for the paths that have no check and never will, because writing one requires knowing the path exists. What a reviewer actually does with a suite like yours is read it as evidence about its author, then go through everything it left out. That is also the difference between this and asking the assistant again. When the AI has been asked ten times and the error is still there, the loop will not break on the eleventh, and whether the AI can be trusted to check its own work is the question sitting underneath. The app somebody else built for you with AI arrives with the same problem plus a suite you did not watch anyone write.
If you want to end up with a suite that catches the next one, that is a different job with its own order of operations: what to test first if you want to build a suite that would have caught it, manual before automated. This page has been about the suite you already have.
Common questions about a green test suite on a broken app
Why do my tests pass but the app doesn’t work?
Because a passing run only re-confirms the checks somebody wrote, and the thing that broke was almost certainly not one of them. On AI-built apps there are three usual reasons at once: no check covers that path, the checks that exist assert what the code already does rather than what you needed, and nothing runs the suite before a deploy anyway. At least 18 of 21 third-party apps in the June and July 2026 cohort had no working test anywhere, so for most owners the honest answer is that the suite was never covering the app in the first place.
Does 100% test coverage mean there are no bugs?
No. Coverage measures which lines of code were executed while the suite ran, not whether anything was verified about them. A check can run a line and assert nothing about the result, and that line counts as covered. It is a measurement of how much of the code the tests walked past, and bugs live in the difference between walking past a line and checking what it decided.
Can I trust tests the AI wrote for me?
Trust them as a description of what the code currently does, and treat the question of whether that is right as still open. When the same session writes the implementation and the checks, the checks inherit the implementation’s assumptions, including its mistakes. They are still worth having, because they will tell you when a later change alters existing behavior. They will not tell you that the behavior they locked in was wrong from the first day.
Should I ask the AI to write more tests?
Only after somebody decides what the checks should be asking. Asking for more checks against the same code produces more agreement with the same code, which is why a suite can grow for weeks without the app getting any more reliable. The useful version is to write down, in plain sentences, the five things that must always be true about your app, then have those turned into checks and confirm each one actually fails when you break the thing it describes.
My app has no tests at all. Is that worse?
It is more honest and roughly as risky. An app with no automated checks tells you nothing, and you know it tells you nothing. A green suite that covers the wrong things tells you nothing while looking like it told you something, and it ends the conversation that would have found the problem. In the 26-app cohort audited in June and July 2026, at least 23 had zero working automated tests, so the no-tests case is the normal one rather than the exception.
How do I know my app actually works before I charge people?
Walk it as a stranger, on the live deployment, with a brand new account, all the way through the action that takes money, then try that same action from an account that never paid. Those two walks catch more real failures on AI-built apps than any number of generated checks, because they exercise the paths a generator was never asked about. Beyond that, the reliable answer is a person reading the code, since the failures that matter most are the ones nobody knew to look for.
Is a green build the same as a green test run?
No, and confusing the two is common. A build going green means the code compiled and packaged without errors. A test run going green means the checks that exist got their expected answers. An app can build perfectly and fail every customer, and several of the audited apps made that worse by disabling their own type and lint checks at build time so broken code would compile anyway. At least 17 of 21 third-party apps in that cohort had no deploy gate at all, so nothing ran either signal before the code reached production.
Not sure what your app needs yet?
See how we follow one real problem from the behavior through the code and decide what should happen next.
