{"slug": "making-documentation-pages-agent-friendly", "title": "Making documentation pages agent friendly", "summary": "A documentation writer is reworking their site's pages to be readable by coding agents, removing JavaScript-driven hidden content because agent fetch tools make an HTTP request, convert HTML to Markdown, and never run JavaScript or click buttons. The Agent-Friendly Documentation Spec sets a 50k-character passing threshold for page size, and in one test by Dachary Carey a MongoDB quick start repeated across 11 drivers and clients produced roughly 258k characters of Markdown, so an agent asking for the Python version never reached the Python tab. Code tabs, expandable text and filterable tables can stay as long as their content is already in the HTML, since Markdown converters ignore the CSS that hides it.", "body_md": "# Making documentation pages agent friendly\n\nThis week one of my primary tasks is to make our documentation site agent friendly. Over the years, writers have incorporated several JS-driven techniques to try to make pages more human friendly. This might involve buttons that you click to reveal hidden content, thereby making the initial page less daunting and only showing content that the reader wants to see when they choose to see it.\n\n## Content agents can and can’t see\n\nThis pattern can still work for agents, but only under one condition. The fetch tools that coding agents use to read pages generally don’t run JavaScript, click buttons, or choose filters. They make an HTTP request, convert the HTML to Markdown, and work from that. (Mikhail Shilkov’s [breakdown of Claude Code’s web tools](https://mikhail.io/2025/10/claude-code-web-tools/) walks through this process step by step, and Anthropic’s [web fetch tool docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool) note that it doesn’t support pages rendered with JavaScript. Browser-based agents can click, but that isn’t how most coding agents read docs.)\n\nAs a result, what matters is where the content lives, not whether JS is involved. If JS only shows and hides content that’s already in the HTML, agents still get that content, because their converters ignore the CSS that hides it. That’s why the Agent-Friendly Documentation Spec [warns about tabbed content](https://agentdocsspec.com/spec/web/content-structure/) serializing into one long document. However, if JS fetches or builds the content after someone clicks, agents never see it. (In Vercel’s [controlled probes](https://vercel.com/kb/guide/make-your-documentation-readable-by-ai-agents), a client that didn’t run JavaScript couldn’t retrieve an answer that existed only in a JavaScript payload. In other words, if the content isn’t in the HTML, it’s not there.)\n\nAs a result, I’m undoing the implementations that keep content out of reach of agents, favoring more conventional stacked sections instead of hidden sections requiring clicks. JS isn’t going away entirely, though. An easy test is to check how your page looks in the markdown mirror view, that is. Is the content easily accessible? If not, fix it.\n\nA few JS conventions still remain, because their content is already in the HTML:\n\n- **Code tabs** – the tabs that sort code into different languages, such as Java | Kotlin | curl, can remain. Reason being, when you choose the MD view, the code in different tabs just stacks nicely on top of each other, with each snippet as its own code block. Short snippets don’t add much length. Tabs cause problems mainly when each tab holds an entire tutorial. In one[test by Dachary Carey](https://dacharycarey.com/2026/02/19/agent-web-fetch-spelunking/) , a MongoDB quick start repeated the same tutorial for 11 drivers and clients, the page’s Markdown source ran about 258k characters, and an agent asking for the Python version never reached the Python tab.\n- **Expandable text** – same reason, as long as the text is in the HTML rather than loaded on click. This content isn’t lost in the MD view. (I even use an expandable button to show a long code example later in this post.)\n- **Filterable tables** – same reason, as long as all the rows are in the HTML and the filter just hides them.\n\nSponsored\n\n## Long pages and truncation\n\nAnother casualty: long pages that exceed 50k characters. The [Agent-Friendly Documentation Spec](https://agentdocsspec.com/spec/web/page-size/) uses 50k characters as its passing threshold for page size. This works out to roughly 8,000 words, which seems like more than enough for most pages. However, some documentation pages, mostly reference material of some kind, can get up to 100k characters or more.\n\nDifferent agents stop reading at different points, so there isn’t a standard cutoff. The [reference MCP fetch server](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) cuts pages off at 5,000 characters by default, while Claude Code [truncates at about 100 KB of text](https://mikhail.io/2025/10/claude-code-web-tools/). The 50k target is meant to fit comfortably within most of these limits.\n\nIf you want to test things out, ask an agent to fetch the page and quote its last line, and then check that quote against the actual page. Don’t just ask the agent whether it read the whole page, since agents’ self-reports about truncation aren’t reliable. Rhyannon Rodriguez reached that conclusion after [testing how Cursor fetches web pages](https://rhyannonjoy.github.io/agent-ecosystem-testing/docs/anysphere-cursor/cursor-interpreted-vs-raw) as part of the Agent Ecosystem Testing project. If the agent can’t quote the last line, or quotes something that isn’t there, you know that it stopped consuming the content before that point. This makes anything below its truncation line invisible to the agent. And what happens when there are invisible spaces, i.e., gaps? AI tools might fill in those gaps with guesses, i.e., hallucinations, which can be wrong.\n\n## How agents fill in the gaps\n\nI’ve seen this fill-in-the-gap behavior firsthand. When writing a strategy doc at work, I was using an agent to add references for the claims. I’d already given the agent quite a few sources, but some parts of the doc didn’t have any sources behind them. The agent wanted to add references for those parts too, and that’s where it went off the rails.\n\nIt started grabbing details here and there from the sources and extrapolating how they could be used as evidence. The connections were plausible in places, but they were clearly a stretch. I realized that this is likely where agents go astray, when they have to fill in the gaps.\n\nTruncated pages put agents in the same situation. Agents often don’t notice when a page has been cut off, so they don’t know what they’re missing. In Rodriguez’s [Cursor tests](https://rhyannonjoy.github.io/agent-ecosystem-testing/docs/anysphere-cursor/cursor-interpreted-vs-raw), the agent treated partial content as complete because the excerpts read coherently on their own. When you leave gaps in your content, agents might fill them in with predictions, drawing on their training data or inferring from what they did read. For a page that’s 75k characters, if the agent consumed only the first 50k characters, it might still answer confidently about something in the last 25k. But when you check that answer against the page, that’s where you start to see factual accuracy take a hit.\n\nThis is also why agents work better in tech comm contexts when you feed them a lot of rich source material. With enough source material, the agent doesn’t have to fill in gaps. Instead, its job is to summarize, reduce, reorganize, and rephrase the material. In other words, it doesn’t have to invent anything.\n\n## Breaking up long pages\n\nTo avoid the fill-in-the-gap-with-guesses syndrome, we have to chunk our content so that agents aren’t truncating their consumption of the page early. This means breaking up long pages. Exactly how we break up the pages depends on the content, but generally this might mean sacrificing some fancy JS mechanisms that let human readers hide, show, and filter content on one long page in a more readable way. In that case, the JS itself isn’t what hurts agents, but the long page that it made workable for humans does.\n\nThis means that yes, in some ways, making content more agent-accessible means tradeoffs for human accessibility. But not in a huge way. The design patterns are simply more plain pages, more first level and second level groupings of pages, rather than all-in-one-pages.\n\n## The Markdown mirror\n\nI mentioned earlier about looking at the markdown mirror of pages. That’s the page that appears when you add `.md` at the end of a page URL, a convention that comes from the [llms.txt proposal](https://llmstxt.org/). (Some sites use `.md.txt` instead, so the convention varies. For example, the Google Maps Platform docs at developers.google.com/maps/documentation serve their Markdown at `.md.txt` URLs, such as [https://developers.google.com/maps/documentation/javascript/overview.md.txt](https://developers.google.com/maps/documentation/javascript/overview.md.txt).) Try it on my site with this page: [https://idratherbewriting.com/blog/make-documentation-agent-friendly.md](https://idratherbewriting.com/blog/make-documentation-agent-friendly.md). What appears is the markdown version of this content.\n\nMarkdown has far fewer characters than HTML, which makes it the syntax of choice for token-constrained agents. (Cloudflare measured one of its own blog posts at 16,180 tokens as HTML and 3,150 tokens as Markdown, according to its [Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) post.) Agents that fetch the regular HTML page convert it to Markdown themselves, but serving your own Markdown lets you control that conversion and leave out navigation and other page chrome.\n\n## Setting up a Markdown mirror with Cloudflare\n\nSetting up a markdown mirror for your site is pretty easy, by the way, but only because of AI. My domain is proxied through Cloudflare, so requests for my pages pass through Cloudflare’s network before they reach my web host. [Cloudflare Workers](https://developers.cloudflare.com/workers/) let you run your own code on those requests. (Cloudflare also offers a no-code option called [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) on its paid plans.) I set up a worker that serves a Markdown version of a page when the URL ends in `.md` or when the request asks for Markdown with an `Accept: text/markdown` header. The markdown-mirror worker code is about 500 lines, so I’ve collapsed it below. The collapsed code is still in the HTML, so it also serves as a test. If you view the [Markdown version of this page](https://idratherbewriting.com/blog/make-documentation-agent-friendly.md), the full code appears there without any clicking.\n\n```\n/**\n * Cloudflare Worker: Markdown Mirror for I'd Rather Be Writing\n * Serves a clean Markdown version of any page when the URL ends in .md\n * or when the request's Accept header asks for text/markdown.\n */\n\nconst SITE_ORIGIN = 'https://idratherbewriting.com';\nconst LLMS_TXT_URL = `${SITE_ORIGIN}/llms.txt`;\nconst MIRROR_USER_AGENT = 'Cloudflare-Markdown-Mirror/1.1';\nconst CACHE_SECONDS = 3600;\n\nexport default {\n  async fetch(request, env, ctx) {\n    const method = request.method.toUpperCase();\n    const url = new URL(request.url);\n\n    const isMdExtension = url.pathname.endsWith('.md');\n    const acceptsMarkdown = (request.headers.get('Accept') || '').toLowerCase().includes('text/markdown');\n\n    // Only GET and HEAD requests that ask for Markdown are handled here\n    if ((method !== 'GET' && method !== 'HEAD') || (!isMdExtension && !acceptsMarkdown)) {\n      return fetch(request);\n    }\n\n    try {\n      return await serveMarkdown(url, method, isMdExtension, ctx);\n    } catch (err) {\n      // Never break the site. If anything goes wrong, fall back to the normal response.\n      console.error('Markdown mirror error:', err && err.stack ? err.stack : err);\n      return fetch(request);\n    }\n  }\n};\n\nasync function serveMarkdown(url, method, isMdExtension, ctx) {\n  const originBase = url.origin.includes('workers.dev') ? SITE_ORIGIN : url.origin;\n\n  // 1. Resolve the path of the HTML page this request maps to\n  let targetPath = url.pathname;\n  if (isMdExtension) {\n    targetPath = targetPath.slice(0, -3); // Strip .md\n    if (targetPath === '' || targetPath.endsWith('/')) {\n      targetPath += 'index.html';\n    }\n  }\n\n  // 2. Check the edge cache. The cache key is separate from the HTML page's URL,\n  //    so a cached Markdown response can never be served to a browser.\n  const cache = caches.default;\n  const cacheKey = new Request(`${originBase}${targetPath}?markdown-mirror=1`, { method: 'GET' });\n  const cached = await cache.match(cacheKey);\n  if (cached) {\n    return method === 'HEAD'\n      ? new Response(null, { status: cached.status, headers: cached.headers })\n      : cached;\n  }\n\n  // 3. Fetch the HTML page from the origin\n  let originRes = await fetchOrigin(new URL(targetPath, originBase));\n\n  // Extensionless .md requests (e.g. /ai/product-skills-docs-first.md) map to .html pages\n  if (originRes.status === 404 && isMdExtension && !/\\.html?$/i.test(targetPath)) {\n    const htmlRes = await fetchOrigin(new URL(`${targetPath}.html`, originBase));\n    if (htmlRes.ok) {\n      originRes = htmlRes;\n    }\n  }\n\n  if (!originRes.ok) {\n    if (!isMdExtension) {\n      return originRes; // Let the origin's own error page through\n    }\n    return textResponse(`Page not found: ${url.pathname}`, originRes.status, method);\n  }\n\n  // Images, PDFs, feeds, llms.txt, and other non-HTML files pass through unchanged\n  const contentType = (originRes.headers.get('Content-Type') || '').toLowerCase();\n  if (!contentType.includes('text/html')) {\n    return originRes;\n  }\n\n  // 4. Convert the HTML to Markdown\n  const canonicalUrl = originRes.url || new URL(targetPath, originBase).toString();\n  const html = await originRes.text();\n  const markdown = convertHtmlToMarkdown(html, canonicalUrl);\n\n  // 5. Return the Markdown and cache it at the edge\n  const headers = {\n    'Content-Type': 'text/markdown; charset=utf-8',\n    'Cache-Control': `public, max-age=${CACHE_SECONDS}`,\n    'Vary': 'Accept',\n    'Link': `<${canonicalUrl}>; rel=\"canonical\"`,\n    'Access-Control-Allow-Origin': '*'\n  };\n  const mdResponse = new Response(markdown, { status: 200, headers });\n  ctx.waitUntil(cache.put(cacheKey, mdResponse.clone()));\n\n  return method === 'HEAD' ? new Response(null, { status: 200, headers }) : mdResponse;\n}\n\nfunction fetchOrigin(originUrl) {\n  return fetch(originUrl.toString(), {\n    headers: {\n      'User-Agent': MIRROR_USER_AGENT,\n      'Accept': 'text/html'\n    }\n  });\n}\n\nfunction textResponse(body, status, method) {\n  return new Response(method === 'HEAD' ? null : body, {\n    status,\n    headers: { 'Content-Type': 'text/plain; charset=utf-8' }\n  });\n}\n\n// ---------------------------------------------------------------------------\n// HTML to Markdown conversion\n// ---------------------------------------------------------------------------\n\nconst PH = String.fromCharCode(0xE000);      // Wraps code placeholders (a private-use character)\nconst INDENT = String.fromCharCode(0xE001);  // Marks intentional indentation that cleanup must keep\nconst BLOCK_TOKEN = (n) => `${PH}B${n}${PH}`;   // Placeholder for a fenced code block\nconst INLINE_TOKEN = (n) => `${PH}I${n}${PH}`;  // Placeholder for an inline code span\n\nfunction convertHtmlToMarkdown(html, canonicalUrl) {\n  // Extract metadata\n  const title = extractTitle(html);\n  const description = getMetaContent(html, 'name', 'description').replace(/\\s+/g, ' ');\n  const dateMatch = html.match(/\"datePublished\":\\s*\"([^\"]+)\"/i);\n  const date = (dateMatch ? dateMatch[1] : getMetaContent(html, 'property', 'article:published_time')).split('T')[0];\n\n  const blocks = [];\n  const inlines = [];\n\n  let content = extractMainContent(html);\n\n  // Remove scripts, styles, comments, and form widgets first, so that markup\n  // inside script strings (like the ads' document.write calls) can't confuse\n  // any of the steps that follow\n  content = content\n    .replace(/<!--[\\s\\S]*?-->/g, '')\n    .replace(/<(script|style|noscript|template|svg|form|button|select|textarea|nav)\\b[\\s\\S]*?<\\/\\1>/gi, '')\n    .replace(/<input\\b[^>]*>/gi, '');\n\n  // Remove page chrome and in-post ads. Ads (div.comicBlock) appear in the middle\n  // of pages, so they're removed here instead of being treated as the end of the page.\n  content = removeElements(content, 'div', classPattern('comicBlock', 'courseProgress', 'lastModifiedDate', 'subheading'));\n  content = removeElements(content, 'ol', classPattern('breadcrumb'));\n  content = removeElements(content, 'ul', /\\bid\\s*=\\s*[\"']markdown-toc[\"']/i);\n  content = removeElements(content, 'p', classPattern('page-meta', 'sponsoredLabel'));\n  content = removeElements(content, 'span', classPattern('pageCategories'));\n\n  // Remove the draft notice but keep the other callouts (Note, Tip, Important, Warning)\n  content = content\n    .replace(/<div\\b[^>]*\\bclass\\s*=\\s*[\"'](?:[^\"']*\\s)?alert(?:\\s[^\"']*)?[\"'][^>]*>\\s*<i\\b[^>]*\\bfa-flask\\b[^>]*>[\\s\\S]*?<\\/div>/gi, '')\n    .replace(/<i\\b[^>]*\\bclass\\s*=\\s*[\"'](?:[^\"']*\\s)?fa(?:\\s[^\"']*)?[\"'][^>]*>\\s*<\\/i>/gi, '')\n    .replace(/<div\\b[^>]*\\bclass\\s*=\\s*[\"'](?:[^\"']*\\s)?(?:alert|summary)(?:\\s[^\"']*)?[\"'][^>]*>([\\s\\S]*?)<\\/div>/gi, '<blockquote>$1</blockquote>');\n\n  // Protect code from the cleanup steps below. Code blocks keep their exact\n  // whitespace and get a fence that's longer than any backtick run inside them.\n  content = content.replace(/<pre\\b([^>]*)>\\s*(?:<code\\b([^>]*)>)?([\\s\\S]*?)(?:<\\/code>\\s*)?<\\/pre>/gi,\n    (match, preAttrs, codeAttrs, code, offset, whole) => {\n      const lang = detectLanguage(`${codeAttrs || ''} ${preAttrs}`, whole.slice(Math.max(0, offset - 300), offset));\n      const text = decodeHtml(code.replace(/<[^>]+>/g, '')).replace(/^\\n+|\\s+$/g, '');\n      const fence = '`'.repeat(Math.max(3, longestRun(text, '`') + 1));\n      return `<div>${BLOCK_TOKEN(blocks.push(`${fence}${lang}\\n${text}\\n${fence}`) - 1)}</div>`;\n    });\n\n  content = content.replace(/<(code|kbd|samp)\\b[^>]*>([\\s\\S]*?)<\\/\\1>/gi, (match, tag, code) => {\n    const text = decodeHtml(code.replace(/<[^>]+>/g, '')).replace(/\\s+/g, ' ');\n    if (!text.trim()) return '';\n    const ticks = '`'.repeat(longestRun(text, '`') + 1);\n    const pad = text.startsWith('`') || text.endsWith('`') ? ' ' : '';\n    return INLINE_TOKEN(inlines.push(`${ticks}${pad}${text}${pad}${ticks}`) - 1);\n  });\n\n  // Whitespace in HTML isn't significant, so collapse it. Line breaks are\n  // re-created from the block-level tags below.\n  content = content.replace(/\\s+/g, ' ');\n\n  // Embedded media becomes links\n  content = content.replace(/<iframe\\b([^>]*)>(?:[\\s\\S]*?<\\/iframe>)?/gi, (match, attrs) => {\n    const src = getAttr(attrs, 'src');\n    if (!src || !/(?:youtube\\.com|youtu\\.be|vimeo\\.com|loom\\.com)/i.test(src)) return '';\n    const label = escapeBrackets(getAttr(attrs, 'title') || 'Video');\n    return `<p>[Video: ${label}](${absoluteUrl(src, canonicalUrl)})</p>`;\n  });\n  content = content.replace(/<(audio|video)\\b([^>]*)>([\\s\\S]*?)<\\/\\1>/gi, (match, tag, attrs, inner) => {\n    const source = inner.match(/<source\\b([^>]*)>/i);\n    const src = getAttr(attrs, 'src') || (source ? getAttr(source[1], 'src') : '');\n    if (!src) return '';\n    const label = tag.toLowerCase() === 'audio' ? 'Audio' : 'Video';\n    return `<p>[${label}](${absoluteUrl(src, canonicalUrl)})</p>`;\n  });\n\n  // Figures and images, with absolute URLs\n  content = content.replace(/<figure\\b[^>]*>([\\s\\S]*?)<\\/figure>/gi, (match, inner) => {\n    const img = inner.match(/<img\\b([^>]*)>/i);\n    if (!img) return `<div>${inner}</div>`; // Not an image figure, so keep its contents\n    const src = getAttr(img[1], 'src');\n    if (!src) return '';\n    const alt = getAttr(img[1], 'alt');\n    const capMatch = inner.match(/<figcaption\\b[^>]*>([\\s\\S]*?)<\\/figcaption>/i);\n    const caption = capMatch ? capMatch[1].trim() : '';\n    const captionText = stripTags(caption);\n    let out = `<p>![${escapeBrackets(alt || captionText || 'Image')}](${absoluteUrl(src, canonicalUrl)})</p>`;\n    if (alt && captionText && captionText !== alt) {\n      out += `<p><em>${caption}</em></p>`;\n    }\n    return out;\n  });\n  content = content.replace(/<img\\b([^>]*)>/gi, (match, attrs) => {\n    const src = getAttr(attrs, 'src');\n    if (!src) return '';\n    return `![${escapeBrackets(getAttr(attrs, 'alt'))}](${absoluteUrl(src, canonicalUrl)})`;\n  });\n\n  // Links, with absolute URLs (same-page #anchors are left as-is)\n  content = content.replace(/<a\\b([^>]*)>([\\s\\S]*?)<\\/a>/gi, (match, attrs, inner) => {\n    const text = inner.replace(/<[^>]+>/g, '').replace(/\\s+/g, ' ').trim();\n    const href = getAttr(attrs, 'href');\n    if (/\\/cdn-cgi\\/l\\/email-protection/i.test(href)) return '(email address hidden)'; // Cloudflare email obfuscation\n    if (!href || /^javascript:/i.test(href)) return text;\n    if (!text) return '';\n    return `[${text}](${absoluteUrl(href, canonicalUrl)})`;\n  });\n\n  // Inline formatting\n  content = content\n    .replace(/<(strong|b)\\b[^>]*>([\\s\\S]*?)<\\/\\1>/gi, (m, tag, inner) => (inner.trim() ? `**${inner.trim()}**` : ''))\n    .replace(/<(em|i|cite)\\b[^>]*>([\\s\\S]*?)<\\/\\1>/gi, (m, tag, inner) => (inner.trim() ? `*${inner.trim()}*` : ''))\n    .replace(/<(del|s|strike)\\b[^>]*>([\\s\\S]*?)<\\/\\1>/gi, (m, tag, inner) => (inner.trim() ? `~~${inner.trim()}~~` : ''))\n    .replace(/<br\\s*\\/?>/gi, '\\n')\n    .replace(/<hr\\b[^>]*>/gi, '\\n\\n---\\n\\n');\n\n  // Tables become Markdown tables instead of runs of cell text\n  content = content.replace(/<table\\b[^>]*>([\\s\\S]*?)<\\/table>/gi, (match, inner) => convertTable(inner));\n\n  // Headings (remove <h1> since the title is already emitted at the top)\n  content = content\n    .replace(/<h1\\b[^>]*>[\\s\\S]*?<\\/h1>/gi, '')\n    .replace(/<h([2-6])\\b[^>]*>([\\s\\S]*?)<\\/h\\1>/gi, (match, level, inner) => {\n      const text = stripTags(inner);\n      return text ? `\\n\\n${'#'.repeat(Number(level))} ${text}\\n\\n` : '';\n    });\n\n  // Definition lists and expandable sections\n  content = content\n    .replace(/<dt\\b[^>]*>([\\s\\S]*?)<\\/dt>/gi, (m, inner) => `\\n\\n**${stripTags(inner)}**\\n`)\n    .replace(/<dd\\b[^>]*>([\\s\\S]*?)<\\/dd>/gi, '$1\\n\\n')\n    .replace(/<\\/?dl\\b[^>]*>/gi, '\\n\\n')\n    .replace(/<summary\\b[^>]*>([\\s\\S]*?)<\\/summary>/gi, (m, inner) => `\\n\\n**${stripTags(inner)}**\\n\\n`)\n    .replace(/<\\/?details\\b[^>]*>/gi, '\\n\\n');\n\n  // Lists (with numbering and nesting) and blockquotes, including callouts inside list steps\n  content = convertListsAndQuotes(content);\n\n  // Paragraphs and other block elements become paragraph breaks\n  content = content\n    .replace(/<\\/?p\\b[^>]*>/gi, '\\n\\n')\n    .replace(/<\\/?(?:div|section|article|header|footer|main|aside|figure|figcaption|center|address)\\b[^>]*>/gi, '\\n\\n')\n    .replace(/<[^>]+>/g, '');\n\n  // Decode HTML entities, then clean up whitespace line by line\n  content = decodeHtml(content)\n    .replace(/\\[email\\sprotected\\]/gi, '(email address hidden)')\n    .split('\\n')\n    .map((line) => line.replace(/\\s+/g, ' ').trim())\n    .join('\\n')\n    .replace(new RegExp(INDENT, 'g'), ' ')\n    .replace(/\\n{3,}/g, '\\n\\n')\n    .trim();\n\n  // Restore code blocks (keeping any list indentation or blockquote prefix) and inline code\n  content = content\n    .replace(new RegExp(`^([ >]*)${PH}B(\\\\d+)${PH}[ ]*$`, 'gm'), (match, prefix, n) =>\n      blocks[Number(n)].split('\\n').map((line) => `${prefix}${line}`).join('\\n'))\n    .replace(new RegExp(`${PH}B(\\\\d+)${PH}`, 'g'), (match, n) => `\\n\\n${blocks[Number(n)]}\\n\\n`)\n    .replace(new RegExp(`${PH}I(\\\\d+)${PH}`, 'g'), (match, n) => inlines[Number(n)]);\n\n  // Build the final document\n  const frontMatter = [\n    '---',\n    `title: ${JSON.stringify(title)}`,\n    date ? `date: ${date}` : null,\n    description ? `description: ${JSON.stringify(description)}` : null,\n    `canonical_url: ${canonicalUrl}`,\n    '---'\n  ].filter(Boolean).join('\\n');\n\n  return [\n    frontMatter,\n    '',\n    `> For AI agents: a documentation index is available at ${LLMS_TXT_URL}. Markdown versions of all pages are available by appending .md to any page URL.`,\n    '',\n    title ? `# ${title}\\n` : null,\n    content,\n    ''\n  ].filter((part) => part !== null).join('\\n');\n}\n\n// Returns the article content: everything inside <div id=\"content\"> up to the\n// \"end page/post content\" comment that closes the article\nfunction extractMainContent(html) {\n  const start = html.match(/<div\\s+id=[\"']content[\"'][^>]*>/i);\n  if (!start) {\n    const body = html.match(/<body\\b[^>]*>([\\s\\S]*)<\\/body>/i);\n    return body ? body[1] : html;\n  }\n  const rest = html.slice(start.index + start[0].length);\n\n  const end = rest.search(/<!--\\s*end (?:page\\/post )?content\\s*-->/i);\n  if (end !== -1) {\n    return rest.slice(0, end);\n  }\n  const fallback = rest.search(/<div\\s+class=[\"'](?:share-buttons|footer|disqus|recent-posts)|<footer\\b/i);\n  return fallback !== -1 ? rest.slice(0, fallback) : rest;\n}\n\nfunction extractTitle(html) {\n  let title = getMetaContent(html, 'property', 'og:title');\n  if (!title) {\n    const match = html.match(/<title\\b[^>]*>([\\s\\S]*?)<\\/title>/i);\n    title = match ? decodeHtml(match[1]) : '';\n  }\n  return title.replace(/\\s*\\|.*$/, '').replace(/\\s+/g, ' ').trim();\n}\n\nfunction getMetaContent(html, attrName, attrValue) {\n  const tags = html.match(/<meta\\b[^>]*>/gi) || [];\n  for (const tag of tags) {\n    if (getAttr(tag, attrName).toLowerCase() === attrValue.toLowerCase()) {\n      return getAttr(tag, 'content').trim();\n    }\n  }\n  return '';\n}\n\n// Reads an attribute value, whether it's double-quoted, single-quoted, or unquoted\nfunction getAttr(attrs, name) {\n  const match = attrs.match(new RegExp(`(?:^|\\\\s)${name}\\\\s*=\\\\s*(?:\"([^\"]*)\"|'([^']*)'|([^\\\\s\"'>]+))`, 'i'));\n  if (!match) return '';\n  const value = match[1] !== undefined ? match[1] : (match[2] !== undefined ? match[2] : match[3]);\n  return decodeHtml(value || '');\n}\n\n// Matches a class attribute that contains one of the given class names as a whole token\nfunction classPattern(...names) {\n  return new RegExp(`\\\\bclass\\\\s*=\\\\s*[\"'](?:[^\"']*\\\\s)?(?:${names.join('|')})(?:\\\\s[^\"']*)?[\"']`, 'i');\n}\n\n// Removes every <tag> whose attributes match attrPattern, including all nested content.\n// If the element's end can't be found, only its opening tag is removed, so content is never lost.\nfunction removeElements(html, tag, attrPattern) {\n  const openRe = new RegExp(`<${tag}\\\\b([^>]*)>`, 'gi');\n  let result = '';\n  let last = 0;\n  let match;\n  while ((match = openRe.exec(html)) !== null) {\n    if (match.index < last || !attrPattern.test(match[1])) continue;\n    const afterOpen = match.index + match[0].length;\n    const end = findElementEnd(html, tag, afterOpen);\n    result += html.slice(last, match.index);\n    last = end === -1 ? afterOpen : end;\n    openRe.lastIndex = last;\n  }\n  return result + html.slice(last);\n}\n\n// Finds the end of the element that opened just before `from`, or -1 if it has no end\nfunction findElementEnd(html, tag, from) {\n  if (tag === 'p') {\n    // A <p> can't contain block elements, and its closing tag is optional in HTML,\n    // so it ends at </p> or at the next block-level tag, whichever comes first\n    const re = /<\\/p\\s*>|<(?:p|div|h[1-6]|ul|ol|dl|table|blockquote|pre|section|figure|hr)\\b/gi;\n    re.lastIndex = from;\n    const match = re.exec(html);\n    if (!match) return -1;\n    return match[0][1] === '/' ? match.index + match[0].length : match.index;\n  }\n\n  // Count nested tags of the same name until the element closes\n  const re = new RegExp(`<(/?)${tag}\\\\b[^>]*>`, 'gi');\n  re.lastIndex = from;\n  let depth = 1;\n  let match;\n  while ((match = re.exec(html)) !== null) {\n    depth += match[1] ? -1 : 1;\n    if (depth === 0) return match.index + match[0].length;\n  }\n  return -1;\n}\n\nfunction detectLanguage(attrs, before) {\n  let match = attrs.match(/\\blanguage-([\\w+#.-]+)/i) || attrs.match(/\\bdata-lang=[\"']([^\"']+)[\"']/i);\n  if (!match) {\n    // Rouge wraps fenced code blocks in <div class=\"language-xyz highlighter-rouge\"><div class=\"highlight\">\n    match = before.match(/<div\\b[^>]*\\blanguage-([\\w+#.-]+)[^>]*>\\s*<div\\b[^>]*\\bhighlight\\b[^>]*>\\s*$/i);\n  }\n  const lang = match ? match[1].toLowerCase() : '';\n  return ['plaintext', 'text', 'none'].includes(lang) ? '' : lang;\n}\n\n// Rows and cells are split on their opening tags, since HTML lets closing tags be omitted\nfunction convertTable(inner) {\n  const rows = [];\n  for (const row of inner.split(/<tr\\b[^>]*>/i).slice(1)) {\n    const cells = row.split(/<t[hd]\\b[^>]*>/i).slice(1)\n      .map((cell) => cell.replace(/<[^>]+>/g, ' ').replace(/\\s+/g, ' ').trim().replace(/\\|/g, '\\\\|'));\n    if (cells.length) rows.push(cells);\n  }\n  if (!rows.length) return '';\n\n  const width = Math.max(...rows.map((r) => r.length));\n  const pad = (r) => r.concat(Array(width - r.length).fill(''));\n  const lines = [pad(rows[0]), Array(width).fill('---'), ...rows.slice(1).map(pad)]\n    .map((r) => `| ${r.join(' | ')} |`);\n  return `\\n\\n${lines.join('\\n')}\\n\\n`;\n}\n\n// Converts lists and blockquotes from the innermost element out, so a list inside a\n// blockquote, or a callout inside a list step, keeps its structure and indentation\nfunction convertListsAndQuotes(html) {\n  const innermost = /<(ul|ol|blockquote)\\b([^>]*)>((?:(?!<\\/?(?:ul|ol|blockquote)\\b)[\\s\\S])*?)<\\/\\1>/i;\n  let guard = 0;\n  while (innermost.test(html) && guard++ < 2000) {\n    html = html.replace(innermost, (match, tag, attrs, inner) =>\n      (tag.toLowerCase() === 'blockquote' ? convertQuote(inner) : convertList(tag, attrs, inner)));\n  }\n  return html;\n}\n\nfunction convertList(tag, attrs, inner) {\n  const ordered = tag.toLowerCase() === 'ol';\n  let n = parseInt(getAttr(attrs, 'start'), 10) || 1;\n  const items = [];\n  // Items are split on their opening tags, since HTML lets </li> be omitted\n  for (const item of inner.split(/<li\\b[^>]*>/i).slice(1)) {\n    const text = blockText(item.replace(/<\\/li\\s*>/gi, ''));\n    if (!text) continue;\n    const marker = ordered ? `${n++}.` : '-';\n    const indent = INDENT.repeat(marker.length + 1);\n    const lines = text.split('\\n');\n    items.push(lines.map((line, i) => (i === 0 ? `${marker} ${line}` : (line ? indent + line : ''))).join('\\n'));\n  }\n  return `\\n\\n${items.join('\\n')}\\n\\n`;\n}\n\nfunction convertQuote(inner) {\n  const lines = blockText(inner).split('\\n');\n  return `\\n\\n${lines.map((line) => (line ? `> ${line}` : '>')).join('\\n')}\\n\\n`;\n}\n\n// Turns a fragment of HTML into lines of text, keeping paragraph and line breaks\nfunction blockText(html) {\n  return html\n    .replace(/<\\/?p\\b[^>]*>/gi, '\\n\\n')\n    .replace(/<\\/?(?:div|section|figure|figcaption)\\b[^>]*>/gi, '\\n\\n')\n    .replace(/<[^>]+>/g, '')\n    .split('\\n')\n    .map((line) => line.trim())\n    .join('\\n')\n    .replace(/\\n{3,}/g, '\\n\\n')\n    .trim();\n}\n\nfunction absoluteUrl(href, baseUrl) {\n  const value = (href || '').trim();\n  if (!value || value.startsWith('#') || /^(?:mailto|tel|data):/i.test(value)) return value;\n  try {\n    return new URL(value, baseUrl).toString()\n      .replace(/ /g, '%20')\n      .replace(/\\(/g, '%28')\n      .replace(/\\)/g, '%29');\n  } catch (err) {\n    return value;\n  }\n}\n\nfunction longestRun(text, char) {\n  let longest = 0;\n  let current = 0;\n  for (const c of text) {\n    current = c === char ? current + 1 : 0;\n    if (current > longest) longest = current;\n  }\n  return longest;\n}\n\nfunction stripTags(html) {\n  return (html || '').replace(/<[^>]+>/g, '').replace(/\\s+/g, ' ').trim();\n}\n\nfunction escapeBrackets(text) {\n  return (text || '').replace(/[[\\]]/g, '\\\\$&');\n}\n\nconst NAMED_ENTITIES = {\n  amp: '&', lt: '<', gt: '>', quot: '\"', apos: \"'\", nbsp: ' ',\n  mdash: '—', ndash: '–', hellip: '…', lsquo: '‘', rsquo: '’', ldquo: '“', rdquo: '”',\n  laquo: '«', raquo: '»', bull: '•', middot: '·', copy: '©', reg: '®', trade: '™',\n  rarr: '→', larr: '←', times: '×', deg: '°', sect: '§', para: '¶'\n};\n\n// Decodes entities in a single pass, so \"&amp;lt;\" correctly becomes \"&lt;\" (not \"<\")\nfunction decodeHtml(str) {\n  if (!str) return '';\n  return str.replace(/&(#x[0-9a-f]+|#\\d+|[a-z][a-z0-9]*);/gi, (match, entity) => {\n    if (entity[0] === '#') {\n      const isHex = entity[1] === 'x' || entity[1] === 'X';\n      const code = isHex ? parseInt(entity.slice(2), 16) : parseInt(entity.slice(1), 10);\n      return code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : match;\n    }\n    const named = NAMED_ENTITIES[entity.toLowerCase()];\n    return named !== undefined ? named : match;\n  });\n}\n```\n\nOf course AI just wrote this. It took a few tries and tests, though, to ensure that all elements were represented. At first, the code choked on my inserted ads. Other issues also had to be ironed out.\n\nLook at your markdown mirror code and make sure that all content is represented as desired, because this is the view agents get when they ask for Markdown. Some agents, such as Claude Code, Cursor, and OpenCode, request Markdown automatically through the `Accept` header ([Checkly](https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/)), and others find the `.md` URLs through links in your [`llms.txt`](https://llmstxt.org/) file. Agents that fetch the regular HTML page use their own converter instead.\n\n## Chunked content for MCP servers\n\nThe markdown mirror view isn’t the view all agents will see, though. There are different agents with different purposes. The markdown mirror is more for agents that fetch pages from your site at run-time for specific coding tasks. Some documentation sites also make their content available through an [MCP](https://modelcontextprotocol.io/) server, which might search an index where the content has been split into smaller passages, or chunks. That’s another view that also merits more discussion.\n\nChunking is a design choice made by whoever builds the index, and splitting at headings is a common approach. For example, [Chroma’s chunking guide](https://docs.trychroma.com/guides/build/chunking) recommends splitting Markdown at h2 headings and then recursively trying inner headings. CT Smith’s [search and chat design](https://docsgoblin.com/blog/26-10-17-jev-and-embeddings.html) on the Docs Goblin blog splits pages at `##` and `###` headings and never cuts a paragraph, list, or code block in half. When the chunker follows your headings, the way you structure a page shapes what an agent retrieves.\n\n## Conclusion\n\nOverall, understanding the view that agents have of content has opened my eyes in a way. I find myself regularly looking at the Markdown version of the page to see just how the content looks. It’s pretty plain, serving up content in Markdown syntax (including Markdown table syntax). I go into more detail on these problems, with sources, in [Making docs accessible to agents](https://idratherbewriting.com/ai/product-skills-agent-friendly-docs.html), a topic in my agent skills course.\n\nWhen I notice issues, they are among the easiest to fix, but you can’t just fix a single instance on your site. Treat each instance as a larger pattern to fix across your entire content.\n\nNote: AI helped me find & summarize resources to corroborate a lot of the assertions in this post.\n\n## About Tom Johnson\n\nI'm an API technical writer based in the Seattle area. On this blog, I write about topics related to technical writing and communication — such as software documentation, API documentation, AI, information architecture, content strategy, writing processes, plain language, tech comm careers, and more. Check out my [API documentation course](https://idratherbewriting.com/learnapidoc/) if you're looking for more info about documenting APIs. Or see my [posts on AI](https://idratherbewriting.com/category-ai/) and [AI course section](https://idratherbewriting.com/ai/index.html) for more on the latest in AI and tech comm.\n\nIf you're a technical writer and want to keep on top of the latest trends in the tech comm, be sure to subscribe to email updates below. You can also [learn more about me](https://idratherbewriting.com/aboutme/) or [contact me](https://idratherbewriting.com/contact/). Finally, note that the opinions I express on my blog are my own points of view, not that of my employer.", "url": "https://wpnews.pro/news/making-documentation-pages-agent-friendly", "canonical_source": "https://idratherbewriting.com/blog/make-documentation-agent-friendly", "published_at": "2026-10-27 18:30:23+00:00", "updated_at": "2026-10-27 18:41:40.513309+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-crawlers", "developer-tools"], "entities": ["Agent-Friendly Documentation Spec", "Mikhail Shilkov", "Anthropic", "Claude Code", "Vercel", "Dachary Carey", "MongoDB", "Model Context Protocol"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/making-documentation-pages-agent-friendly", "markdown": "https://wpnews.pro/news/making-documentation-pages-agent-friendly.md", "text": "https://wpnews.pro/news/making-documentation-pages-agent-friendly.txt", "jsonld": "https://wpnews.pro/news/making-documentation-pages-agent-friendly.jsonld"}}