llms.txt in 2026: who actually reads it (and why we shipped one anyway)

llms.txt is a proposed web standard — from Jeremy Howard of Answer.AI, September 2024 — that puts a curated, markdown index of your site at /llms.txt so language models get a clean map instead of crawling noisy HTML: a one-paragraph summary of what the product is, then sections of links with one-line descriptions (optionally /llms-full.txt with the docs inlined whole).

It’s also having the strangest adoption curve in web standards: roughly one in ten sites now ships one, while the crawlers it was designed for almost universally ignore it. Most writing about it picks a side — “essential for AI visibility!” or “cargo cult!” — and both are wrong in instructive ways. Here’s the honest state of it, and why we shipped ours anyway.

The uncomfortable data

If you ship llms.txt expecting AI search traffic, the evidence is against you:

• Server-log studies show major AI crawlers essentially never request it. One analysis of 500 million AI-bot visits over 90 days found 408 direct llms.txt fetches — statistical zero.
• Google is on record twice: John Mueller noting no AI service claims to use it, and Gary Illyes confirming Google doesn’t support it and isn’t planning to.
• In a Search Engine Land experiment, 8 of 9 sites saw no measurable traffic change after implementing it.

So: not a rankings play, not an AI-search play. Anyone selling it as one is selling you something.

Who actually reads it

Coding agents and AI-powered IDEs. When an agent is pointed at a documentation site — “integrate this SDK,” “call this API” — the common playbook is to check for /llms.txt, use it to find the relevant page, and pull only that. For developer-facing products, the file has a real, current audience; it just isn’t the audience the hype describes. That’s also where adoption is concentrated: developer-tool companies ship it; general marketing sites mostly shouldn’t bother yet.

Why we shipped one anyway

For Prufa the calculus is unusually clear, because agents are not a hypothetical audience for us — they’re a first-class user. An AI agent that just built a web app is exactly who should be calling our API to verify it. Our llms.txt exists for the agent that lands on prufa.dev with that job:

• It leads with what an agent needs before calling us: the free audit endpoint, idempotency keys on every mutating call, API versioning.
• It links the agent surface first — SKILL.md, the OpenAPI contract, the public BeaconEvent and flow-spec specs — and the human landing page last, as “optional.”
• It cost about thirty minutes, because the hard part already existed: the README-as-agent-landing-page, versioned specs, and OpenAPI docs. llms.txt is an index, and an index is only as good as what it points to.

That last point is the real lesson, and it’s why the “essential vs. cargo cult” framing misses. The durable work is making your product legible to agents at all — machine-readable contracts, stable URLs, docs that are markdown-clean. llms.txt is the cheap table of contents on top. Shipped in that order, it’s a half-hour bet with genuine upside and no downside. Shipped instead of that work, it’s a decoration.

If you ship one: the 30-minute version

1. One H1 with your product name, one blockquote summarizing what it does and who it’s for — written for a model deciding whether you’re relevant, not for a human scrolling.
2. A few short sections of links (## Docs, ## API, …), each link with a one-line description. Absolute URLs. Point at markdown or clean HTML.
3. Put the things an agent must know before acting (auth, rate limits, idempotency, versioning) as plain bullet points near the top — don’t make the model infer them.
4. Keep it short and current. A field audit of 30 real llms.txt files found the same anti-patterns repeating: dumping the entire sitemap, marketing copy instead of facts, stale links.
5. Don’t expect traffic. Expect the occasional coding agent to get your integration right on the first try. That’s the actual prize today.

Ours is at prufa.dev/llms.txt — and if your AI built your site, the free audit is how it checks its own work.