Making documentation pages agent friendly A documentation writer is reworking their site's pages to be readable by coding agents, removing JavaScript-driven hidden content because agent fetch tools make an HTTP request, convert HTML to Markdown, and never run JavaScript or click buttons. The Agent-Friendly Documentation Spec sets a 50k-character passing threshold for page size, and in one test by Dachary Carey a MongoDB quick start repeated across 11 drivers and clients produced roughly 258k characters of Markdown, so an agent asking for the Python version never reached the Python tab. Code tabs, expandable text and filterable tables can stay as long as their content is already in the HTML, since Markdown converters ignore the CSS that hides it. Making documentation pages agent friendly This week one of my primary tasks is to make our documentation site agent friendly. Over the years, writers have incorporated several JS-driven techniques to try to make pages more human friendly. This might involve buttons that you click to reveal hidden content, thereby making the initial page less daunting and only showing content that the reader wants to see when they choose to see it. Content agents can and can’t see This pattern can still work for agents, but only under one condition. The fetch tools that coding agents use to read pages generally don’t run JavaScript, click buttons, or choose filters. They make an HTTP request, convert the HTML to Markdown, and work from that. Mikhail Shilkov’s breakdown of Claude Code’s web tools https://mikhail.io/2025/10/claude-code-web-tools/ walks through this process step by step, and Anthropic’s web fetch tool docs https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool note that it doesn’t support pages rendered with JavaScript. Browser-based agents can click, but that isn’t how most coding agents read docs. As a result, what matters is where the content lives, not whether JS is involved. If JS only shows and hides content that’s already in the HTML, agents still get that content, because their converters ignore the CSS that hides it. That’s why the Agent-Friendly Documentation Spec warns about tabbed content https://agentdocsspec.com/spec/web/content-structure/ serializing into one long document. However, if JS fetches or builds the content after someone clicks, agents never see it. In Vercel’s controlled probes https://vercel.com/kb/guide/make-your-documentation-readable-by-ai-agents , a client that didn’t run JavaScript couldn’t retrieve an answer that existed only in a JavaScript payload. In other words, if the content isn’t in the HTML, it’s not there. As a result, I’m undoing the implementations that keep content out of reach of agents, favoring more conventional stacked sections instead of hidden sections requiring clicks. JS isn’t going away entirely, though. An easy test is to check how your page looks in the markdown mirror view, that is. Is the content easily accessible? If not, fix it. A few JS conventions still remain, because their content is already in the HTML: - Code tabs – the tabs that sort code into different languages, such as Java | Kotlin | curl, can remain. Reason being, when you choose the MD view, the code in different tabs just stacks nicely on top of each other, with each snippet as its own code block. Short snippets don’t add much length. Tabs cause problems mainly when each tab holds an entire tutorial. In one test by Dachary Carey https://dacharycarey.com/2026/02/19/agent-web-fetch-spelunking/ , a MongoDB quick start repeated the same tutorial for 11 drivers and clients, the page’s Markdown source ran about 258k characters, and an agent asking for the Python version never reached the Python tab. - Expandable text – same reason, as long as the text is in the HTML rather than loaded on click. This content isn’t lost in the MD view. I even use an expandable button to show a long code example later in this post. - Filterable tables – same reason, as long as all the rows are in the HTML and the filter just hides them. Sponsored Long pages and truncation Another casualty: long pages that exceed 50k characters. The Agent-Friendly Documentation Spec https://agentdocsspec.com/spec/web/page-size/ uses 50k characters as its passing threshold for page size. This works out to roughly 8,000 words, which seems like more than enough for most pages. However, some documentation pages, mostly reference material of some kind, can get up to 100k characters or more. Different agents stop reading at different points, so there isn’t a standard cutoff. The reference MCP fetch server https://github.com/modelcontextprotocol/servers/tree/main/src/fetch cuts pages off at 5,000 characters by default, while Claude Code truncates at about 100 KB of text https://mikhail.io/2025/10/claude-code-web-tools/ . The 50k target is meant to fit comfortably within most of these limits. If you want to test things out, ask an agent to fetch the page and quote its last line, and then check that quote against the actual page. Don’t just ask the agent whether it read the whole page, since agents’ self-reports about truncation aren’t reliable. Rhyannon Rodriguez reached that conclusion after testing how Cursor fetches web pages https://rhyannonjoy.github.io/agent-ecosystem-testing/docs/anysphere-cursor/cursor-interpreted-vs-raw as part of the Agent Ecosystem Testing project. If the agent can’t quote the last line, or quotes something that isn’t there, you know that it stopped consuming the content before that point. This makes anything below its truncation line invisible to the agent. And what happens when there are invisible spaces, i.e., gaps? AI tools might fill in those gaps with guesses, i.e., hallucinations, which can be wrong. How agents fill in the gaps I’ve seen this fill-in-the-gap behavior firsthand. When writing a strategy doc at work, I was using an agent to add references for the claims. I’d already given the agent quite a few sources, but some parts of the doc didn’t have any sources behind them. The agent wanted to add references for those parts too, and that’s where it went off the rails. It started grabbing details here and there from the sources and extrapolating how they could be used as evidence. The connections were plausible in places, but they were clearly a stretch. I realized that this is likely where agents go astray, when they have to fill in the gaps. Truncated pages put agents in the same situation. Agents often don’t notice when a page has been cut off, so they don’t know what they’re missing. In Rodriguez’s Cursor tests https://rhyannonjoy.github.io/agent-ecosystem-testing/docs/anysphere-cursor/cursor-interpreted-vs-raw , the agent treated partial content as complete because the excerpts read coherently on their own. When you leave gaps in your content, agents might fill them in with predictions, drawing on their training data or inferring from what they did read. For a page that’s 75k characters, if the agent consumed only the first 50k characters, it might still answer confidently about something in the last 25k. But when you check that answer against the page, that’s where you start to see factual accuracy take a hit. This is also why agents work better in tech comm contexts when you feed them a lot of rich source material. With enough source material, the agent doesn’t have to fill in gaps. Instead, its job is to summarize, reduce, reorganize, and rephrase the material. In other words, it doesn’t have to invent anything. Breaking up long pages To avoid the fill-in-the-gap-with-guesses syndrome, we have to chunk our content so that agents aren’t truncating their consumption of the page early. This means breaking up long pages. Exactly how we break up the pages depends on the content, but generally this might mean sacrificing some fancy JS mechanisms that let human readers hide, show, and filter content on one long page in a more readable way. In that case, the JS itself isn’t what hurts agents, but the long page that it made workable for humans does. This means that yes, in some ways, making content more agent-accessible means tradeoffs for human accessibility. But not in a huge way. The design patterns are simply more plain pages, more first level and second level groupings of pages, rather than all-in-one-pages. The Markdown mirror I mentioned earlier about looking at the markdown mirror of pages. That’s the page that appears when you add .md at the end of a page URL, a convention that comes from the llms.txt proposal https://llmstxt.org/ . Some sites use .md.txt instead, so the convention varies. For example, the Google Maps Platform docs at developers.google.com/maps/documentation serve their Markdown at .md.txt URLs, such as https://developers.google.com/maps/documentation/javascript/overview.md.txt https://developers.google.com/maps/documentation/javascript/overview.md.txt . Try it on my site with this page: https://idratherbewriting.com/blog/make-documentation-agent-friendly.md https://idratherbewriting.com/blog/make-documentation-agent-friendly.md . What appears is the markdown version of this content. Markdown has far fewer characters than HTML, which makes it the syntax of choice for token-constrained agents. Cloudflare measured one of its own blog posts at 16,180 tokens as HTML and 3,150 tokens as Markdown, according to its Markdown for Agents https://blog.cloudflare.com/markdown-for-agents/ post. Agents that fetch the regular HTML page convert it to Markdown themselves, but serving your own Markdown lets you control that conversion and leave out navigation and other page chrome. Setting up a Markdown mirror with Cloudflare Setting up a markdown mirror for your site is pretty easy, by the way, but only because of AI. My domain is proxied through Cloudflare, so requests for my pages pass through Cloudflare’s network before they reach my web host. Cloudflare Workers https://developers.cloudflare.com/workers/ let you run your own code on those requests. Cloudflare also offers a no-code option called Markdown for Agents https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ on its paid plans. I set up a worker that serves a Markdown version of a page when the URL ends in .md or when the request asks for Markdown with an Accept: text/markdown header. The markdown-mirror worker code is about 500 lines, so I’ve collapsed it below. The collapsed code is still in the HTML, so it also serves as a test. If you view the Markdown version of this page https://idratherbewriting.com/blog/make-documentation-agent-friendly.md , the full code appears there without any clicking. / Cloudflare Worker: Markdown Mirror for I'd Rather Be Writing Serves a clean Markdown version of any page when the URL ends in .md or when the request's Accept header asks for text/markdown. / const SITE ORIGIN = 'https://idratherbewriting.com'; const LLMS TXT URL = ${SITE ORIGIN}/llms.txt ; const MIRROR USER AGENT = 'Cloudflare-Markdown-Mirror/1.1'; const CACHE SECONDS = 3600; export default { async fetch request, env, ctx { const method = request.method.toUpperCase ; const url = new URL request.url ; const isMdExtension = url.pathname.endsWith '.md' ; const acceptsMarkdown = request.headers.get 'Accept' || '' .toLowerCase .includes 'text/markdown' ; // Only GET and HEAD requests that ask for Markdown are handled here if method == 'GET' && method == 'HEAD' || isMdExtension && acceptsMarkdown { return fetch request ; } try { return await serveMarkdown url, method, isMdExtension, ctx ; } catch err { // Never break the site. If anything goes wrong, fall back to the normal response. console.error 'Markdown mirror error:', err && err.stack ? err.stack : err ; return fetch request ; } } }; async function serveMarkdown url, method, isMdExtension, ctx { const originBase = url.origin.includes 'workers.dev' ? SITE ORIGIN : url.origin; // 1. Resolve the path of the HTML page this request maps to let targetPath = url.pathname; if isMdExtension { targetPath = targetPath.slice 0, -3 ; // Strip .md if targetPath === '' || targetPath.endsWith '/' { targetPath += 'index.html'; } } // 2. Check the edge cache. The cache key is separate from the HTML page's URL, // so a cached Markdown response can never be served to a browser. const cache = caches.default; const cacheKey = new Request ${originBase}${targetPath}?markdown-mirror=1 , { method: 'GET' } ; const cached = await cache.match cacheKey ; if cached { return method === 'HEAD' ? new Response null, { status: cached.status, headers: cached.headers } : cached; } // 3. Fetch the HTML page from the origin let originRes = await fetchOrigin new URL targetPath, originBase ; // Extensionless .md requests e.g. /ai/product-skills-docs-first.md map to .html pages if originRes.status === 404 && isMdExtension && /\.html?$/i.test targetPath { const htmlRes = await fetchOrigin new URL ${targetPath}.html , originBase ; if htmlRes.ok { originRes = htmlRes; } } if originRes.ok { if isMdExtension { return originRes; // Let the origin's own error page through } return textResponse Page not found: ${url.pathname} , originRes.status, method ; } // Images, PDFs, feeds, llms.txt, and other non-HTML files pass through unchanged const contentType = originRes.headers.get 'Content-Type' || '' .toLowerCase ; if contentType.includes 'text/html' { return originRes; } // 4. Convert the HTML to Markdown const canonicalUrl = originRes.url || new URL targetPath, originBase .toString ; const html = await originRes.text ; const markdown = convertHtmlToMarkdown html, canonicalUrl ; // 5. Return the Markdown and cache it at the edge const headers = { 'Content-Type': 'text/markdown; charset=utf-8', 'Cache-Control': public, max-age=${CACHE SECONDS} , 'Vary': 'Accept', 'Link': <${canonicalUrl} ; rel="canonical" , 'Access-Control-Allow-Origin': ' ' }; const mdResponse = new Response markdown, { status: 200, headers } ; ctx.waitUntil cache.put cacheKey, mdResponse.clone ; return method === 'HEAD' ? new Response null, { status: 200, headers } : mdResponse; } function fetchOrigin originUrl { return fetch originUrl.toString , { headers: { 'User-Agent': MIRROR USER AGENT, 'Accept': 'text/html' } } ; } function textResponse body, status, method { return new Response method === 'HEAD' ? null : body, { status, headers: { 'Content-Type': 'text/plain; charset=utf-8' } } ; } // --------------------------------------------------------------------------- // HTML to Markdown conversion // --------------------------------------------------------------------------- const PH = String.fromCharCode 0xE000 ; // Wraps code placeholders a private-use character const INDENT = String.fromCharCode 0xE001 ; // Marks intentional indentation that cleanup must keep const BLOCK TOKEN = n = ${PH}B${n}${PH} ; // Placeholder for a fenced code block const INLINE TOKEN = n = ${PH}I${n}${PH} ; // Placeholder for an inline code span function convertHtmlToMarkdown html, canonicalUrl { // Extract metadata const title = extractTitle html ; const description = getMetaContent html, 'name', 'description' .replace /\s+/g, ' ' ; const dateMatch = html.match /"datePublished":\s " ^" + "/i ; const date = dateMatch ? dateMatch 1 : getMetaContent html, 'property', 'article:published time' .split 'T' 0 ; const blocks = ; const inlines = ; let content = extractMainContent html ; // Remove scripts, styles, comments, and form widgets first, so that markup // inside script strings like the ads' document.write calls can't confuse // any of the steps that follow content = content .replace /< -- \s\S ?-- /g, '' .replace /< script|style|noscript|template|svg|form|button|select|textarea|nav \b \s\S ?<\/\1 /gi, '' .replace /