AI testing false positives: 4 of our 6 findings were wrong

A false positive in AI testing is a finding the agent reports as a defect when the application was working correctly — and the honest way to talk about them is with your own numbers. On 3 July 2026 we ran our chaos agent against a live production CRM, with the owner’s permission and their credentials. It emitted 6 findings. We labeled every one by hand: 2 were real, 2 were outright false positives, and 2 described the app correctly refusing an unauthorized action. Four of six should never have reached a customer’s report.

Both false positives came from our highest-confidence tier. That is the part worth publishing, because the fix it forced is architectural rather than a matter of better prompting: an LLM in a QA agent should never be allowed to decide a question that plain code can observe directly. This is what our agent got wrong, the guards that now kill each class, and the test that fails our build if we regress.

What our agent got wrong, finding by finding

The run explored the app as a “confused newbie” persona with a 20-step cap. Here is the whole labeled set, in the language of the labels file that ships in our test corpus:

#	What was reported	Verdict
1	Console error: 401 rejection creating a task	Environmental — the permission wall
2	Judge verdict: “the user’s task failed”	Environmental — the same 401, counted twice
3	Judge verdict: a filled field didn’t hold the typed text	False positive (verified tier)
4	Layout: two elements overlapping	False positive (verified tier)
5	Contrast: ADD button at 2.13:1	Real (advisory)
6	Contrast: “Sign in with Google” on mobile	Real (advisory)

Three distinct mistakes are hiding in that table.

Finding 3 — the model overruled the ground truth. The agent typed Call John about proposal into a field. The field then contained exactly Call John about proposal. The LLM judge, comparing the rendered page against what it believed the expected value was, decided the step had failed. Nothing was broken; the judge was comparing against a stale expectation it had constructed itself.

Finding 4 — geometry without meaning. Our layout detector measures element rectangles and reports overlaps, which is a real class of bug: a button covered by a banner is a button nobody can click. Here it found a logo glyph sitting entirely inside its own logo button and reported the two as colliding. They are one control. The math was right and the conclusion was wrong.

Findings 1 and 2 — the app worked, twice. The test account was deliberately restricted. When it tried to create a task on an account it had no rights to, the API returned 401 and the UI showed a permission message. That is authorization functioning exactly as designed. Our agent reported it as a defect — and then reported it again, because a console-error detector and the semantic judge both witnessed the same event and each produced its own finding.

Why do AI testing tools report false positives?

The industry’s answer to this question is remarkably uniform. The advice you will find is to write more stable selectors, replace fixed waits with condition-based ones, let the model learn from historical runs, keep a human in the loop, or add a “reviewer agent” that reads the screenshot and network log before a finding is surfaced. Testing vendors describe the failure mode precisely — an agent marks a step failed because the actual message is close to the expected message but not identical, and a human later decides the behavior was fine — and then prescribe more training.

Every one of those is a tuning knob. Tuning moves the rate; it doesn’t bound it. And the reviewer-agent pattern has an obvious structural weakness: it is an LLM adjudicating an LLM, so its own errors are correlated with the errors it is meant to catch.

Our three mistakes above suggest a different taxonomy, one about authority rather than accuracy:

1. A model judged a question that had an observable answer. “Does the field contain the text we typed” is not a matter of interpretation. Whenever a model is allowed to rule on something plain code can read, you have imported the model’s error rate for free.
2. A correct refusal was read as a failure. Permission walls, paywalls, validation rejections, rate limits — an exploratory agent will trip all of them, especially one deliberately supplied with a low-privilege account. Failure of an action is not a defect in the app.
3. One event produced several findings. Independent detectors watching the same step each report it. The customer counts findings, not events, and concludes the app is four times as broken as it is.

The rule: a finding is a falsifiable claim, a deterministic oracle, and evidence

That taxonomy produced a single design rule that now governs every check our agent can emit. If a claim has an observable ground truth — a field value, an element count, an HTTP status, a URL change, a console message — then plain code decides it and the model never gets a vote. If it doesn’t, the claim is advisory by default, and may only be promoted to a verified tier under narrow conditions: it must carry a browser evidence chain and a screenshot, and it must survive an adversarial check that can only lower confidence.

The rule is enforced as a registry rather than a convention. Every check identifier the agent can produce is classified in one table — deterministic, semantic-judge, or advisory-only — with the oracle that decides it named on the same line. A contract test fails the build if any check is missing from that table, so a new detector cannot ship without someone stating, in code, what would prove it wrong. The classification is the interesting part of the table: console errors, HTTP failures, double submissions, stuck spinners, dead ends, layout overflow, broken images, occluded targets, layout shift — all decided by plain code, zero LLM on the decision path. The genuinely semantic questions are a short list, and they’re the ones under the strictest scrutiny.

This is the same invariant we described in agentic testing vs scripted testing — the LLM navigates, plain code verifies — but the registry is what turns a stated principle into something a build can check.

The three guards that kill the three classes

The fill oracle. A “fill” step with no submit intent is now never shown to the judge at all. Whether the field received the text is answered by reading the field. That single gate eliminates the entire class that produced finding 3 — not by making the judge smarter, but by removing its jurisdiction.

The benign-overlap guard. Rect intersection still runs in the browser; the decision moved to Python, behind four deliberately narrow rules. An overlap is dismissed only when the two elements share their nearest interactive ancestor, when the intersection’s hit target belongs to both, when one rect sits at least 95% inside the other and the inner one is icon-shaped, or when a label is wrapping its own input. Narrow is the point: two genuinely distinct overlapping controls must still be reported, so the guard is written to under-suppress.

The expected-failure taxonomy. When the response status and the visible copy together show the app refusing correctly, the event becomes a neutral note on the report — not a finding — and any bug-tier findings the same event produced are rerouted into that note. That single mechanism fixes both finding 1 and finding 2: the wall is recorded once, as a wall.

The conservatism in that classifier matters more than its cleverness, so it’s worth being explicit about the three ways it deliberately refuses to fire:

• A 5xx never classifies as an expected wall. The server breaking is always a bug, whatever the page says.
• Every class except rate-limiting requires both a matching status and corroborating user-visible copy.
• A 4xx with no visible feedback is deliberately not classified. An app that rejects your action and tells you nothing has a real UX problem, so that finding stays in the normal path.

A guard that suppresses too eagerly is worse than the false positives it removes, because its failures are invisible.

Verification that can only take confidence away

Two passes run after the findings exist, and both are constrained in the same direction.

The first is an adversarial verifier: a skeptical reviewer whose instructions are to refute each verified finding — is this a real product bug, or is it a third-party analytics error, an expected message, a cosmetic nitpick? It is bounded to the ten most severe findings per run, structural facts about the run itself are exempt from refutation (whether the agent hit an auth wall isn’t a refutable product claim), and refuted findings are demoted to advisory rather than deleted, so nothing disappears silently. Critically, it can only down-tier. It is a skeptic, not a second author, and any error or unparseable reply leaves the finding exactly as it was.

The second is reproduction: a finding should replay or it isn’t verified. Each candidate verified finding whose recorded path exports a valid flow spec is re-executed in a fresh browser session by a deterministic executor using the recorded selectors — no model involved. If the same failure signal appears, the finding is confirmed. If the replay is clean, it drops to advisory. If it replays into an expected wall, it’s reclassified as one.

The fourth outcome is the one that took discipline to get right. If the replay couldn’t execute cleanly — a selector no longer resolves, the executor stalls — the result is inconclusive and the tier is untouched. An executor hiccup is not evidence against a finding. It would have been easy to treat “didn’t reproduce” as “wasn’t real” and watch our precision numbers improve; it would also have quietly destroyed recall, which is the failure mode nobody notices until a customer’s real bug never made the report.

That same discipline forces an honest disclosure: recorded paths don’t include the deterministic pre-LLM login step, which runs outside the recorder. So a replay of a finding on an authenticated page usually stalls at the login gate and returns inconclusive. Replay is conclusive for public-surface findings today, and we’d rather say so than imply a guarantee that holds only on marketing pages. (Getting an agent through a login at all turned out to be its own investigation.)

Measuring it: a labeled corpus and a floor the build enforces

None of the above means anything without a number attached, so the run described at the top of this article is now a test fixture. Its report and its hand-written labels sit in our repository as the seed case of a precision corpus, and a harness replays those recorded findings through the current production guard functions — importing the real modules, never a reimplementation, so the harness cannot drift from the runner. There is no browser, no network, and no model call; the judge’s original verdicts are taken as recorded, and what’s measured is whether today’s plain-code guards keep or suppress them.

Two metrics, defined in the harness itself:

• Precision — of what we still show the customer, how much is real?
• Recall — of what is real, how much do we still show? A suppressed valid finding is a recall failure, and the guards are scored on it just as hard.

Both floors are set to 1.0. On the seed case, the guards must remove all four bad findings and keep both real contrast findings — and the test asserts attribution, not just totals: the console error and the judge verdict must die specifically to the expected-wall guard, the field-value verdict specifically to the fill oracle, the overlap specifically to the benign-overlap guard. Totals can be right by accident. Attribution can’t.

The suite also includes a test that deliberately breaks us. It disables the fill oracle, replays the corpus, and asserts that the false positive comes back and precision drops below the floor. A regression harness that has never been shown to fail is a harness nobody should trust.

The whole thing runs offline in 0.09 seconds — six tests — and has its own job in our CI pipeline, so a change that makes our agent noisier fails the build rather than reaching a customer. We are, in the least glamorous possible way, running QA on our QA.

Honest limits

The corpus is one case. Six findings, one application, one run. A precision floor of 1.0 across a single labeled case is a regression tripwire, not a quality score — it proves today’s guards still clean a set of mistakes we’ve already seen, and proves nothing about mistakes we haven’t. The test file says as much next to the constants: raise the corpus size before ever lowering the floor. Anyone quoting a product-wide false-positive rate from a sample of six is selling something; we’d rather publish the six.

Replay is conclusive only on public surfaces today, for the auth reason above.

The adversarial verifier is best-effort. Model errors, quota exhaustion, or an unparseable reply leave tiers unchanged. That’s the right default — an unavailable skeptic shouldn’t silently downgrade real findings — but it means the pass is not a guarantee on any individual run.

And where the alternative genuinely wins: a hand-written Playwright or Cypress assertion has no false-positive class of this kind at all. It checks precisely what you told it to check, and if it fails, something you cared about changed. All of this machinery exists because exploratory chaos testing buys you coverage you never had to write — including bugs on paths nobody thought to script — and the bill for that coverage is arriving as noise. Scripted tests pay a different bill: you only ever find what you predicted. If you know exactly what to assert, assert it. Our job is to be useful on everything else, without wasting your afternoon.

What this buys you

When you read a Prufa report, the tier on a finding is a claim about how it was decided. Verified means plain code observed it, or a semantic judgment survived an evidence chain, a skeptic that could only lower it, and — on public surfaces — a fresh-session replay. Advisory means it’s worth your eye but we won’t pretend it’s proven. Expected walls are recorded as notes, so an app correctly refusing an unauthorized action never inflates the count.

None of that makes our agent incapable of being wrong. It makes our being wrong the kind of thing that gets caught by a test rather than by a customer. If you want to see what the output looks like on your own app, the chaos mode is here, and the per-flow evidence chain is described here.