Agentic testing vs scripted testing: can you trust an LLM?

The honest answer to “agentic testing vs scripted testing” is that the choice is a false one. The paradigm worth trusting isn’t either — it’s the hybrid: let an LLM-backed agent navigate the app, and let plain, deterministic code verify the result. Agentic tests get their bad reputation from setups where the same model both drives the test and grades it. Split those two jobs and the flakiness goes with it.

That distinction is the whole article. Scripted tests are deterministic but brittle; agentic tests are resilient but only as trustworthy as whatever decides pass or fail. Put the model in charge of navigation and plain code in charge of the verdict, and you get the resilience without surrendering the determinism. Below is why the usual framing is wrong, where the non-determinism actually lives, and the one question that tells trustworthy tools apart.

Agentic testing vs scripted testing: the real difference

A scripted test is an explicit sequence: click this selector, type into that field, assert this element exists. It is perfectly repeatable — run it twice, get the same result — which is exactly the property a test needs. Its weakness is that it knows nothing. Rename a button, move a form across two screens, interpose a cookie banner, and the script breaks even though the app is fine. Teams pay for that brittleness in maintenance: the endless healing of locators that the vendor literature on agentic testing is right to complain about.

An agentic test hands the navigation to an LLM-backed agent. You describe the goal — “sign up with this email, reach the dashboard” — and the agent figures out the clicks, absorbing UI ambiguity the way a human tester would. That resilience is real and valuable. The trap is what happens next: most agentic tools also let the model decide whether the flow succeeded. That is where determinism dies. A language model is persuasive and occasionally, confidently wrong; a persuasive tester that’s wrong is worse than no tester.

So the real axis isn’t “agentic vs scripted.” It’s who decides pass or fail — a model, or plain code. Scripted testing answers that question with code by default. Agentic testing can answer it either way. The good answer is the same in both cases.

Why “agentic testing is flaky” is only half true

The category has a pre-seeded narrative — deterministic is good, agentic is flaky — and it’s worth engaging honestly rather than waving away, because it’s half right. QA Wolf positions deterministic code as the reliable core, and they’re correct that verification must be deterministic. We agree completely; it’s our architecture.

Where the “flaky” charge falls apart is the diagnosis. Flakiness has two sources, and only one is about agents:

1. Unstable environments — timing, async state, shared fixtures, a slow network. This makes scripted tests flaky too. It’s not an agentic problem; it’s a testing problem.
2. Letting the model judge — asking the LLM “did this pass?” and trusting the answer. The output is non-deterministic by construction, so the verdict wobbles run to run.

Source (2) is what people actually mean when they call agentic testing flaky, and it’s entirely avoidable. Even the vendors selling the agentic dream concede the navigation trade-off is mostly about speed, not correctness — an agent that must analyze, plan, then act is slower than a script that already knows what to click, and well-built agent systems already distinguish genuine failures from flakes across runs. The fix for verdict-flakiness isn’t to abandon agents. It’s to stop letting the agent keep score.

The hybrid: the LLM navigates, plain code verifies

This is the shape we build on, and the one we’d recommend to anyone evaluating an AI tester. We state it as a hard invariant: the LLM navigates, plain code verifies; LLM-judged UX findings are advisory-tier only (docs/PLAN.md). Concretely, that splits a test run into two layers that never blur:

Navigation (the model’s job). An LLM-backed agent drives a real browser through a few primitives — act, observe, extract. It finds the “Sign up” button, deals with the relocated field, walks the multi-screen form. This is the part that genuinely needs a model, and it’s where the resilience comes from.

Verification (plain code’s job). Underneath the agent, a plain-code harness owns the browser session and records what actually happened: every network request and response, console output, URLs, cookies, screenshots. After the run, deterministic checks grade that recording against a spec you approved in advance. An expect on the final URL is a string comparison against a recorded value; a submit either returned a success status or a 500, and the finding carries the real response code as evidence. Run the spec twice, get the same verdict — the property a monitor must have before its alerts mean anything. (The concrete walkthrough is in how Prufa verifies a signup flow.)

The separation is structural, not a prompt instruction. There’s no model in the verification loop to drift, no temperature to tune, nothing to convince. When a run claims it found a signup flow, plain code re-checks the entry element in the live DOM before the claim counts (FlowDetection.verified_in_dom in backend/service/checks/schema.py). And the model’s own failures can’t masquerade as findings: “no analytics fired” is only emitted when capture provably worked — the debugger attached and real traffic was observed (CaptureMeta.capture_trustworthy) — so a broken run can’t report a healthy app as broken.

When the model does have an opinion — “this error copy seems misleading” — it ships in a separate advisory tier, labeled an opinion and never phrased as broken (FindingTier in the same file). Verified findings and advisory findings never share a list. That two-tier split is what lets even an unscripted chaos-testing agent wander your app and still produce findings you can trust. That same division — the LLM navigates, deterministic code grades — is exactly what Prufa’s MCP server exposes to an outside agent, so a coding agent can call QA the way it calls any other tool and still get a verdict computed by plain code.

There’s a pricing consequence worth naming, because it shows the determinism is load-bearing and not a slogan: once a flow is confirmed with pinned selectors, replaying it runs the same plain-code path with zero LLM calls — verified in backend/tests/test_flows/test_flow_pipeline.py. Deterministic verification isn’t just more trustworthy; it’s cheaper to repeat.

Are AI tests deterministic? Two different questions

“Are AI tests deterministic” is the query everyone types, and it hides two unrelated questions that the search results constantly conflate:

• Can you test a non-deterministic AI system? This is about grading an LLM or chatbot whose output varies. The standard answer — and what most “deterministic AI testing” results actually discuss — is to wrap the model in deterministic code assertions and average behavior over multiple runs. Even Anthropic’s guidance on evaluating agents describes a deterministic-checks-plus-advisory-judge hybrid. That’s a real problem, but it’s not yours unless you ship an AI feature.
• Is an AI-driven test run repeatable? This is the question a founder testing a normal web app is actually asking. And the answer is: yes, if the grading is deterministic. The navigation can vary (the agent might take a slightly different path to the same button) while the verdict stays fixed, because the verdict is computed by code from the recording, not narrated by the model.

Confusing the two is how the “AI testing is non-deterministic, therefore untrustworthy” myth survives. The model’s navigation being non-deterministic is fine — desirable, even, since that’s what survives a UI change. The verdict being non-deterministic is the only thing that would matter, and the hybrid architecture removes it.

What to ask when you evaluate an AI QA tool

You don’t need to adjudicate the paradigm war. You need to ask each tool one question:

When it says “this works,” who checked — the model, or plain code?

If the answer is “our agent verifies it,” push: verify how? Does a deterministic check read the actual DOM, network response, and cookies, or does the model look at the screen and form an opinion? Does the tool tell you when it was blocked or couldn’t reach a step, or does it render a confident all-clear regardless? A tool you can trust draws the line between what it proved and what it merely thinks — and shows you which is which.

That’s the difference that actually matters in “agentic testing vs scripted testing.” Agentic navigation buys you resilience to change. Deterministic verification buys you a verdict you can wake someone up over. The tools worth using give you both; the ones to avoid give you a model’s confidence dressed up as a result. For the longer argument about why QA needs this shape now, see why Prufa exists — or run a free audit on your own site (just a URL, no signup) and read which findings come back verified and which come back advisory. If the navigator is itself an agent, you can give Claude Code a QA tool and let it do the driving while Prufa’s plain code keeps score. The same verified-not-assumed standard applies on the write side too: an agent retrying one of our own API calls gets a deterministic replay, not a guess — see idempotency keys for AI agents.

FAQ

Are AI-generated tests reliable?

Partly. AI raises coverage fast, but it tends to generate assertions that capture what the code currently does rather than what it should do — so a test written against a buggy app happily locks in the bug. Studies of AI-generated test cases find accuracy drops several points when the code under test contains defects. The fix is the same as for any test: the pass/fail oracle must be deterministic and written against the intended behavior, not inferred by the same model that wrote the code.

Is agentic testing flaky?

Not inherently. Flakiness comes from two sources: unstable environments (timing, shared state) and letting the language model decide pass or fail. The first affects scripted tests too. The second is specific to naive agentic setups and is avoidable: if plain code grades the recorded outcome and the model only handles navigation, the verdict is repeatable. Agentic navigation plus deterministic verification is less flaky than brittle selector scripts, not more.

Can AI replace QA testers?

No — the industry consensus is augment, not replace. AI handles volume: walking known flows, re-checking them on every deploy, surfacing mechanical findings. Humans hold the parts a model is bad at: exploratory testing, business-logic judgment, and deciding whether “works” is actually correct for your domain. A tool that claims to replace your QA function is overselling; one that clears the repetitive table so humans spend time on judgment is doing the real job.

How does an AI agent deterministically verify a test passed?

By separating navigation from grading. The agent drives the browser (non-deterministic), but a plain-code harness records every network request, console line, URL, and cookie, then deterministic checks grade that recording against an approved spec. The verdict is a string comparison or a status-code check, not a model opinion — so the same run produces the same answer. Any AI-judged read (a UX impression) ships in a separate advisory tier, never as a pass/fail.