{"slug": "designing-docs-for-ai-agents", "title": "Designing Docs for AI Agents", "summary": "AI agents now outnumber human readers of technical documentation by thousands to one, according to Cloudflare data showing Anthropic's crawlers fetched 38,000 pages for every human visitor it referred in July 2025. Jamdesk, a docs platform, argues that traditional documentation designed for human attention must be inverted for agents, who read everything and require explicit, standalone content served as markdown rather than HTML to avoid token waste. The company recommends using llms.txt, per-page .md endpoints, and an MCP server to create multiple entry points into the same content.", "body_md": "# Designing Docs for AI Agents\n\nLast week, Ben Swerdlow at Freestyle published [Designing APIs for Agents](https://www.freestyle.sh/blog/opinion/designing-apis-for-agents), arguing that the API design rules we spent twenty years learning are backwards for AI agents. Agents read everything, so explicitness beats convenience, and precise errors beat forgiving defaults.\n\nHe's right, and the same inversion has already hit documentation.\n\nMost docs teams are just coming to realize this.\n\nWe run a [docs platform](https://www.jamdesk.com/?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents), so we know a few things about docs. Here's what's we've learned: write everything down, make every page stand alone, serve markdown instead of chrome, and treat your error messages as first-class content.\n\nKey Takeaways\n\n- In July 2025,\n[Cloudflare]measured Anthropic crawling roughly 38,000 pages for every visitor it referred back. Docs are being read at machine scale, by readers who never see your site.- The same docs page costs 31x more bytes as HTML than as markdown. Chrome is a token tax on every agent that reads you.\n- Progressive disclosure, the core of human docs design, actively hurts agents.\n- You don't need separate docs. You need more entry points into the same content:\n`llms.txt`\n\n, per-page`.md`\n\nendpoints, and an MCP server.\n\n## Your Reader are Now AI Agents\n\nDocumentation traffic is shifting from people to machines, fast. From May 2024 to May 2025, AI and search crawler traffic grew 18%, with OpenAI's GPTBot alone up 305% ([Cloudflare](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), 2025).\n\nNeed more convincing? In July 2025, Anthropic's crawlers fetched about 38,000 pages for every one human visitor they referred back; OpenAI's ratio was around 1,100 to one ([Cloudflare](https://blog.cloudflare.com/crawlers-click-ai-bots-training/), 2025).\n\nMeanwhile the humans moved to asking chat/AI agents instead of their own research. By 2025, Stack Overflow was receiving about as few questions per month as it did in 2009, the year it launched ([The Pragmatic Engineer](https://blog.pragmaticengineer.com/stack-overflow-is-almost-dead/), 2025). Did the developers stop having questions? Nope, they started asking assistants, and the assistants ask your docs.\n\nSo the question isn't whether to design docs for agents. Agents are already your highest-volume audience.\n\nThe question is whether they can use what they find.\n\n## The Playbook We All Learned\n\nGood docs for humans follow certain rules, which can be summarized as \"[Don't make me think](https://www.amazon.com/dp/0321965515)\". For example, start with a quickstart guide, everyone loves help videos, and hide the complex flags until the reader needs them. You also want to have everything well organized in a side nav so I can quickly find what I'm looking for.\n\nEvery one of those rules exists because human attention is scarce. Your users are busy, get bored in minutes, and leave the moment they feel lost (or assume this isn't what they are looking for). Traditionally, yours docs serve two purposes: technical/informational data and marketing. The former is self-evident, but marketing you might have not considered.\n\nWe wrote a whole post on [writing documentation developers actually read](https://www.jamdesk.com/blog/how-to-write-documentation-that-developers-actually-read?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents), and if you compress it to one idea, it's attention management: respect the reader's time, surface the likely next step, get out of the way.\n\nDesign for the reader's scarcest resource, in other words. For humans, that's attention.\n\n## Agents Break Every Assumption\n\nAn agent's scarce resource is context tokens: what it can afford to fetch and hold while answering. Attention never enters into it.\n\nWhen agents arrive at your site, they aren't navigating, click though your sidebar or typing in your search box, they are fetching. An agent lands on one URL, cold, and either finds the answer there or fetches again.\n\nAgents read everything you give them, instantly. Swerdlow's phrase for APIs was \"agents can read our entire docs in one sitting.\" Hiding complexity from them doesn't reduce confusion as it does for people.\n\nAnd agents arrive opinionated. Their training data contains your 2025 docs, your deprecated auth flow, your old parameter names. Live documentation is the correction mechanism for those stale priors. If the current truth isn't written down and fetchable, the agent uses the old version.\n\nOne caution before you swing the other way: reading everything isn't free. In July 2025, Chroma tested 18 frontier models and found every single one degraded as input length grew, a failure mode they named context rot ([Chroma Research](https://research.trychroma.com/context-rot), 2025), so don't make one giant page of all your docs.\n\n## Use Your Words\n\nGive agents the words without the website by proving them with markdown (md file). We measured our own quickstart both ways, and you can rerun this today:\n\n``` bash\n$ curl -sLo /dev/null -w \"%{size_download} bytes\\n\" https://jamdesk.com/docs/quickstart\n225527 bytes\n\n$ curl -sLo /dev/null -w \"%{size_download} bytes\\n\" https://jamdesk.com/docs/quickstart.md\n7246 bytes\n```\n\nThat is 31x fewer bytes, and another example is Vercel ran the same test on their own pages in 2026 and measured [500 KB of HTML against 3 KB of markdown](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation).\n\nAll that extra load (218 K) are scripts, styles, nav, and hydration payload - what makes the web so pretty. But agents don't need or want this, since they will spend tokens downloading it all and then throw it away.\n\n*Measured 2026-07-16. Both URLs are live if you want to check our math.*\n\nAnd to really hammer the point home, [we scored seven docs platforms for AI-friendliness](https://www.jamdesk.com/blog/ai-friendly-docs-platforms-scored?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents) and one served 1,049,919 bytes of HTML containing 1,279 bytes of text because the content only existed after JavaScript ran. To a crawler doing plain HTTP, those docs are a loading spinner.\n\nThe fix is a `.md`\n\nmarkdown mirror of every page. You create the same URL plus `.md`\n\n, plain markdown. [Stripe ships this](https://docs.stripe.com/llms.txt) across their entire docs site. [So do we](https://www.jamdesk.com/docs/ai/markdown-source?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents), on every Jamdesk site, automatically.\n\nServer-render your HTML too. But give agents the door that doesn't cost 31x.\n\n## Be Explicit\n\nSwerdlow's harshest API rule is \"defaults are bad\" and the magic that helps humans makes agents guess. The docs equivalent is implicit knowledge is bad, [explicit knowledge is good](https://www.merriam-webster.com/grammar/usage-of-explicit-vs-implicit).\n\nWhat do I mean by this? Remember, the AI agent might only be fetching a single page or even just a section of it. So you don't want to make the AI have to guess, assume, or have to figure out contradictions.\n\nFor example:\n\n**No relative references.**\"As mentioned above\" and \"see the previous section\" are dead weight to an agent who fetched one page. Link to the actual page instead. Also, this is a good SEO practice.**Full context per page.** Restate which product, which version, which API. The agent may have fetched nothing else.**One name per concept.** If the same thing is a \"project\" on one page, a \"site\" on another, and a \"workspace\" in the dashboard, an agent treats them as three things. Swerdlow calls this staying in distribution. Tech writers call it terminology discipline and consider it a best practice.**And date things.**\"As of the 2026-07-28 protocol revision...\" lets an agent weigh your page against what it half-remembers from training.\n\n## Errors Are Documentation\n\nAn agent debugs by searching fop the error. If it hits a failure, the agent does a search with the error message as a query string. Pretty sweet if your docs are the result!\n\nA good idea is to document the error path with the same priority as your tutorials. A page per error, or a catalog, with the verbatim message as the heading:\n\n```\n## Error: `DNS_VERIFICATION_FAILED`\n\n**What it means:** We couldn't find the TXT record proving you own this domain.\n\n**Why it happens:** The record hasn't propagated (wait up to 48h), or it was\nadded to the wrong zone (check the apex, not the subdomain).\n\n**Fix:** `dig TXT _jamdesk-verify.yourdomain.com` should return your token.\nIf it doesn't, re-add the record from Settings → Domains → Verify.\n```\n\nAs Swerdlow put it for APIs: errors are one of the best surfaces an agent has for finding the happy path. Your docs are where that surface lives.\n\n## Examples are Important\n\nFor human readers, a code sample is a teaching aid. For agents, it's a template that gets instantiated verbatim, at scale. Agents pattern-match against examples more strongly than against prose, which means a subtly wrong sample has a large impact.\n\nSome good guidelines on examples for AI agents:\n\n**Runnable as printed.** Real endpoints, real field names, no`<your-thing-here>`\n\ninside strings that look copy-pasteable.**Complete.** Include the imports, the auth header, the response shape. An agent won't infer the missing 20% the way a senior dev would; it will invent it.**Current.** A deprecated example is worse than no example, because it's*evidence*for the stale behavior the agent already believes.\n\n## The Discovery Stack: llms.txt, .md, MCP\n\nEverything above assumes the agent found the right page. That's the job of three pieces of infrastructure, one of which is genuinely controversial.\n\n*Three entry points into the same content.*\n\n** llms.txt** is a markdown index at your site root: what exists, where, one fetch. By May 2026, 36,120 sites published one, up 8.8x in a year (\n\n[PPC Land](https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/), 2026).\n\nBut do the AI agents read the llms.txt file? In June 2026, [Ahrefs measured 97% of those files receiving zero requests](https://ahrefs.com/blog/what-is-llms-txt/) in a month. Google's John Mueller had [compared the file to the keywords meta tag](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/) a year earlier (2025), and [defenders](https://searchengineland.com/no-llms-txt-is-not-the-new-meta-keywords-458199) have argued with him since.\n\nOur take: Mueller is right about background crawlers and wrong about what else matters. Agentic fetchers (a Claude Code session, a Cursor index, anything told to go read the docs) use it as a map when it exists, because it's the cheapest possible answer to \"what pages should I fetch?\"\n\n[Stripe ships one](https://docs.stripe.com/llms.txt) with instructions for LLM agents embedded in it. When it's [auto-generated from your nav](https://www.jamdesk.com/docs/ai/llms-txt?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents), like ours, the cost is zero. Zero-cost insurance doesn't need a consensus.\n\n**Per-page .md endpoints**. As discussed above (oops, we shouldn't be doing this), the token and speed savings are measurable.\n\n**An MCP server** turns your docs from pages into tools: `searchDocs`\n\n, `getPage`\n\n. Every Jamdesk site [exposes one at /_mcp](https://www.jamdesk.com/docs/ai/mcp-server?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents), no key required. One command makes any of our customers' docs readable from inside Claude Code:\n\n```\nclaude mcp add --transport http jamdesk-docs https://jamdesk.com/docs/_mcp\n```\n\nOne last trick: put instructions for agents *inside the content*, where every reader lands. The top of our own quickstart reads:\n\n```\n> **For AI agents:** the complete documentation index is at [llms.txt](/docs/llms.txt).\n> Append `.md` to any page URL for its markdown version.\n```\n\nAny agent that lands on any page now knows about both entry points.\n\n## Summary\n\nYour docs were already [your most important marketing channel](https://www.jamdesk.com/blog/api-docs-marketing-tool?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents) back when only humans read them. Now they're your interface to every AI assistant your users work inside, so you should focus on making them the most AI-friendly docs possible. We also built [AI Score](https://www.jamdesk.com/docs/ai/ai-score?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents) to automate exactly this audit, and [scored seven platforms with it](https://www.jamdesk.com/blog/ai-friendly-docs-platforms-scored?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents) if you want a baseline.\n\nOr, if you'd rather let someone else do it for you, every Jamdesk site comes with the whole stack by default: `.md`\n\nendpoints, `llms.txt`\n\n, `llms-full.txt`\n\n, and an MCP server, generated from the docs you already have.\n\n[Deploy one in five minutes](https://www.jamdesk.com/docs/quickstart?utm_source=blog&utm_medium=article&utm_campaign=designing-docs-for-ai-agents), then curl your own quickstart with `.md`\n\non the end.", "url": "https://wpnews.pro/news/designing-docs-for-ai-agents", "canonical_source": "https://www.jamdesk.com/blog/designing-docs-for-ai-agents", "published_at": "2026-07-30 15:42:44+00:00", "updated_at": "2026-07-30 15:52:14.376756+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["Cloudflare", "Anthropic", "OpenAI", "Jamdesk", "Ben Swerdlow", "Freestyle", "Stack Overflow", "The Pragmatic Engineer"], "alternates": {"html": "https://wpnews.pro/news/designing-docs-for-ai-agents", "markdown": "https://wpnews.pro/news/designing-docs-for-ai-agents.md", "text": "https://wpnews.pro/news/designing-docs-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/designing-docs-for-ai-agents.jsonld"}}