Introducing AEO Bench

A new benchmark called AEO Bench, built by an unnamed team, measures how AI agents actually interact with site-side readiness techniques such as llms.txt files, sitemaps, and markdown content negotiation, rather than relying on untested advice. The benchmark uses a controlled 40-page fictional outdoor-gear retailer with realistic noise, 36 gradable questions, and pre-registered methodology to test whether models discover and use these techniques. Initial results, detailed in a separate post titled "Nobody Reads llms.txt (Yet)," surprised the researchers in both directions.

Back to Blog /blog Introducing AEO Bench: Measuring Agent Readiness Instead of Guessing Introducing AEO Bench: Measuring Agent Readiness Instead of Guessing AI agents are becoming the web's newest audience. They arrive without eyes, without patience for your hero animation, and increasingly with a budget: every page they read costs somebody tokens. A cottage industry of advice has appeared about how to welcome them, with Cloudflare's agent-readiness proposal as the most concrete menu so far: llms.txt files, sitemaps, markdown content negotiation, hidden directives, machine-readable service cards. The advice sounds right. Most of it has never been measured. We have been here before. Our Barkup Bench series spent two months testing plausible-sounding rules for LLM agents and watching a healthy fraction of them fail, backfire, or turn out to be doing nothing. The lesson we keep relearning is that sounding right is not evidence, and the fix is always the same: pre-register a study, run it against real models, publish whatever comes out. So we built a second benchmark and pointed it at the agent-readiness question. How it works AEO Bench tests site-side techniques against a controlled website: a fictional 40-page outdoor-gear retailer that exists only as a committed data file and a deterministic generator. Every page has realistic noise navigation, promos, cookie banners so that token savings get measured against genuine clutter. The site runs with real HTTP semantics, so content negotiation, redirects, and discovery files behave exactly as they would in production. Agents get a question, the site's root URL, and a fetch tool. The tool's description never mentions any technique, which makes discovery part of the measurement: if a model requests markdown, it found the pathway itself. Thirty-six seeded questions have mechanically gradable answers, including eight questions the site deliberately cannot answer, where the only correct response is saying so. Everything is pre-registered by commit before the first scored run: the fixture, the tasks, the graders, the hypotheses, and the spending limits. Results publish as found, corrections included. The whole harness is open source https://github.com/kevinpeckham/aeo-bench . Why we're doing this Partly because our clients are starting to ask, and we would rather answer with data than with vendor blog posts. Partly because the questions are genuinely open: nobody seems to know whether any model actually reads llms.txt. And partly because the discipline is the product. When we recommend an agent-readiness technique to a client, we want the sentence after the recommendation to be a measurement. Study 1 is already complete, and its results surprised us in both directions. That story is its own post: Nobody Reads llms.txt Yet /blog/nobody-reads-llms-txt . The full dashboard, with every study as it lands, lives at /research/aeo-bench /research/aeo-bench .