Designing Docs for AI Agents AI agents now outnumber human readers of technical documentation by thousands to one, according to Cloudflare data showing Anthropic's crawlers fetched 38,000 pages for every human visitor it referred in July 2025. Jamdesk, a docs platform, argues that traditional documentation designed for human attention must be inverted for agents, who read everything and require explicit, standalone content served as markdown rather than HTML to avoid token waste. The company recommends using llms.txt, per-page .md endpoints, and an MCP server to create multiple entry points into the same content. Designing Docs for AI Agents Last week, Ben Swerdlow at Freestyle published Designing APIs for Agents https://www.freestyle.sh/blog/opinion/designing-apis-for-agents , arguing that the API design rules we spent twenty years learning are backwards for AI agents. Agents read everything, so explicitness beats convenience, and precise errors beat forgiving defaults. He's right, and the same inversion has already hit documentation. Most docs teams are just coming to realize this. We run a docs platform https://www.jamdesk.com/?utm source=blog&utm medium=article&utm campaign=designing-docs-for-ai-agents , so we know a few things about docs. Here's what's we've learned: write everything down, make every page stand alone, serve markdown instead of chrome, and treat your error messages as first-class content. Key Takeaways - In July 2025, Cloudflare measured Anthropic crawling roughly 38,000 pages for every visitor it referred back. Docs are being read at machine scale, by readers who never see your site.- The same docs page costs 31x more bytes as HTML than as markdown. Chrome is a token tax on every agent that reads you. - Progressive disclosure, the core of human docs design, actively hurts agents. - You don't need separate docs. You need more entry points into the same content: llms.txt , per-page .md endpoints, and an MCP server. Your Reader are Now AI Agents Documentation traffic is shifting from people to machines, fast. From May 2024 to May 2025, AI and search crawler traffic grew 18%, with OpenAI's GPTBot alone up 305% Cloudflare https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/ , 2025 . Need more convincing? In July 2025, Anthropic's crawlers fetched about 38,000 pages for every one human visitor they referred back; OpenAI's ratio was around 1,100 to one Cloudflare https://blog.cloudflare.com/crawlers-click-ai-bots-training/ , 2025 . Meanwhile the humans moved to asking chat/AI agents instead of their own research. By 2025, Stack Overflow was receiving about as few questions per month as it did in 2009, the year it launched The Pragmatic Engineer https://blog.pragmaticengineer.com/stack-overflow-is-almost-dead/ , 2025 . Did the developers stop having questions? Nope, they started asking assistants, and the assistants ask your docs. So the question isn't whether to design docs for agents. Agents are already your highest-volume audience. The question is whether they can use what they find. The Playbook We All Learned Good docs for humans follow certain rules, which can be summarized as " Don't make me think https://www.amazon.com/dp/0321965515 ". For example, start with a quickstart guide, everyone loves help videos, and hide the complex flags until the reader needs them. You also want to have everything well organized in a side nav so I can quickly find what I'm looking for. Every one of those rules exists because human attention is scarce. Your users are busy, get bored in minutes, and leave the moment they feel lost or assume this isn't what they are looking for . Traditionally, yours docs serve two purposes: technical/informational data and marketing. The former is self-evident, but marketing you might have not considered. We wrote a whole post on writing documentation developers actually read https://www.jamdesk.com/blog/how-to-write-documentation-that-developers-actually-read?utm source=blog&utm medium=article&utm campaign=designing-docs-for-ai-agents , and if you compress it to one idea, it's attention management: respect the reader's time, surface the likely next step, get out of the way. Design for the reader's scarcest resource, in other words. For humans, that's attention. Agents Break Every Assumption An agent's scarce resource is context tokens: what it can afford to fetch and hold while answering. Attention never enters into it. When agents arrive at your site, they aren't navigating, click though your sidebar or typing in your search box, they are fetching. An agent lands on one URL, cold, and either finds the answer there or fetches again. Agents read everything you give them, instantly. Swerdlow's phrase for APIs was "agents can read our entire docs in one sitting." Hiding complexity from them doesn't reduce confusion as it does for people. And agents arrive opinionated. Their training data contains your 2025 docs, your deprecated auth flow, your old parameter names. Live documentation is the correction mechanism for those stale priors. If the current truth isn't written down and fetchable, the agent uses the old version. One caution before you swing the other way: reading everything isn't free. In July 2025, Chroma tested 18 frontier models and found every single one degraded as input length grew, a failure mode they named context rot Chroma Research https://research.trychroma.com/context-rot , 2025 , so don't make one giant page of all your docs. Use Your Words Give agents the words without the website by proving them with markdown md file . We measured our own quickstart both ways, and you can rerun this today: bash $ curl -sLo /dev/null -w "%{size download} bytes\n" https://jamdesk.com/docs/quickstart 225527 bytes $ curl -sLo /dev/null -w "%{size download} bytes\n" https://jamdesk.com/docs/quickstart.md 7246 bytes That is 31x fewer bytes, and another example is Vercel ran the same test on their own pages in 2026 and measured 500 KB of HTML against 3 KB of markdown https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation . All that extra load 218 K are scripts, styles, nav, and hydration payload - what makes the web so pretty. But agents don't need or want this, since they will spend tokens downloading it all and then throw it away. Measured 2026-07-16. Both URLs are live if you want to check our math. And to really hammer the point home, we scored seven docs platforms for AI-friendliness https://www.jamdesk.com/blog/ai-friendly-docs-platforms-scored?utm source=blog&utm medium=article&utm campaign=designing-docs-for-ai-agents and one served 1,049,919 bytes of HTML containing 1,279 bytes of text because the content only existed after JavaScript ran. To a crawler doing plain HTTP, those docs are a loading spinner. The fix is a .md markdown mirror of every page. You create the same URL plus .md , plain markdown. Stripe ships this https://docs.stripe.com/llms.txt across their entire docs site. So do we https://www.jamdesk.com/docs/ai/markdown-source?utm source=blog&utm medium=article&utm campaign=designing-docs-for-ai-agents , on every Jamdesk site, automatically. Server-render your HTML too. But give agents the door that doesn't cost 31x. Be Explicit Swerdlow's harshest API rule is "defaults are bad" and the magic that helps humans makes agents guess. The docs equivalent is implicit knowledge is bad, explicit knowledge is good https://www.merriam-webster.com/grammar/usage-of-explicit-vs-implicit . What do I mean by this? Remember, the AI agent might only be fetching a single page or even just a section of it. So you don't want to make the AI have to guess, assume, or have to figure out contradictions. For example: No relative references. "As mentioned above" and "see the previous section" are dead weight to an agent who fetched one page. Link to the actual page instead. Also, this is a good SEO practice. Full context per page. Restate which product, which version, which API. The agent may have fetched nothing else. One name per concept. If the same thing is a "project" on one page, a "site" on another, and a "workspace" in the dashboard, an agent treats them as three things. Swerdlow calls this staying in distribution. Tech writers call it terminology discipline and consider it a best practice. And date things. "As of the 2026-07-28 protocol revision..." lets an agent weigh your page against what it half-remembers from training. Errors Are Documentation An agent debugs by searching fop the error. If it hits a failure, the agent does a search with the error message as a query string. Pretty sweet if your docs are the result A good idea is to document the error path with the same priority as your tutorials. A page per error, or a catalog, with the verbatim message as the heading: Error: DNS VERIFICATION FAILED What it means: We couldn't find the TXT record proving you own this domain. Why it happens: The record hasn't propagated wait up to 48h , or it was added to the wrong zone check the apex, not the subdomain . Fix: dig TXT jamdesk-verify.yourdomain.com should return your token. If it doesn't, re-add the record from Settings → Domains → Verify. As Swerdlow put it for APIs: errors are one of the best surfaces an agent has for finding the happy path. Your docs are where that surface lives. Examples are Important For human readers, a code sample is a teaching aid. For agents, it's a template that gets instantiated verbatim, at scale. Agents pattern-match against examples more strongly than against prose, which means a subtly wrong sample has a large impact. Some good guidelines on examples for AI agents: Runnable as printed. Real endpoints, real field names, no