Chaos testing for web apps: what it is, and what an LLM adds

Chaos testing for web apps is the practice of driving your app with unscripted, adversarial behaviour — instead of a fixed test path — to surface bugs that no scripted test was written to catch. Where an end-to-end test walks a route you already chose, a chaos test lets a tester loose to misbehave: fill forms with garbage, double-click submit, mash the back button, wander into pages you forgot existed. The bugs it finds are the ones you didn’t know to look for.

Chaos testing, monkey testing, and the missing middle

The idea is old. Monkey testing — firing random clicks and keystrokes at an app until something falls over — has been around for decades. It’s cheap and it genuinely finds crashes, but it has two weaknesses: it’s undirected (a random walk wastes most of its actions on nothing interesting), and it can’t tell you what broke beyond “it threw an exception.”

The missing middle is a tester that behaves like a plausible bad user rather than a random number generator, and a grader that knows the difference between a real failure and noise. That middle is what an LLM-backed agent now makes practical.

What an LLM adds — and what it must not touch

An LLM is good at exactly the part monkey testing was bad at: looking at an unfamiliar screen and deciding what a frustrated human would try next. Point it at a page it has never seen and it will find the signup form, type an obviously invalid email, submit it twice, and follow a broken link — directed misbehaviour, not noise.

The danger is letting the same model decide whether the result is a bug. Language models are persuasive, and a persuasive tester that is wrong is worse than no tester. So the load-bearing rule in a trustworthy chaos tester is a hard split: the LLM navigates, plain code verifies. The model proposes the next action; deterministic detectors — not the model — decide what counts as broken. A 500 response, an uncaught JavaScript exception, a form that accepted invalid input, a layout wider than the viewport: those are measured off the live page and shipped as verified findings with evidence. Anything the model merely suspects ships in a separate advisory tier, labelled as opinion. The two never mix. (We walk through that architecture on a concrete run in how Prufa verifies a signup flow.)

The two hard problems: safety and false positives

Turning a chaos agent loose on a real app raises two questions that decide whether it’s usable.

Safety. An undirected tester will eventually click “Delete account” or “Place order.” The answer is to deny mutations by default: run dry-run and abort every non-GET request at the network layer, so a destructive click becomes a “would have mutated” finding instead of an action, and real payment instruments are never used. Writing to real data is opt-in, per-domain, and capped.

False positives. A chaos tester that cries wolf is quickly ignored. The same discipline that makes findings trustworthy — plain code, not the model, grades them — also means the detectors must be tuned conservatively: a slow-but-valid spinner, an intentional empty state, or an expected validation error must not mint a verified finding. We learned this on ourselves; see what happened when we ran chaos QA on our own site, including the two detector false positives it surfaced and how we fixed them.

Where chaos testing fits

Chaos testing doesn’t replace your end-to-end suite — it covers a different risk. Scripted tests defend the journeys you already know matter and catch regressions on those exact paths. Chaos testing discovers the failures you never wrote a test for. The two compose well: let the chaos agent explore, and when it reproduces a real bug, promote that reproduction into a permanent, deterministic flow so the bug can never come back unnoticed.

For the longer argument on why QA needs this shape now that agents both build and break software, read why Prufa exists. To see the chaos modality itself, the Gremlin mode page covers the personas, the detectors, and the safety model — or run a free audit to see the deterministic engine on your own URL first. If you’d rather drive it programmatically, you can run chaos QA from your agent via MCP or kick off a run from the API.