{"slug": "give-ai-agents-the-markdown-they-actually-want", "title": "Give AI Agents the Markdown They Actually Want", "summary": "Fastly released an open-source JavaScript service for its Compute platform that automatically converts web pages into clean Markdown for AI crawlers, reducing bandwidth and parsing overhead. The service, deployable in about 200 lines of code, detects AI user-agents and serves Markdown instead of HTML, addressing the fact that bots account for 49% of web requests. This approach improves efficiency for LLM training pipelines and retrieval systems that natively consume Markdown.", "body_md": "AI crawlers are going to ask for your pages whether you're ready for them or not. Today they get HTML, same as any browser, and they spend CPU cycles stripping your nav and footer to find the article underneath. This post walks through a small Fastly Compute service that meets them in the middle: normal requests still get your site, while agents get a clean Markdown version of the same content.\n\nWe can easily accomplish this with about 200 lines of JavaScript, which you can find in the [ repo here](https://github.com/fastly/html-2-md). You can skim the pipeline section to see the shape, or clone and deploy if you want to get there faster.\n\n## Why This Matters\n\nOur own [ Security Research report](https://learn.fastly.com/Security-Threat-Insights-Report) found that bots account for 49% of requests. The vast majority is unwanted traffic, and verified AI is only a sliver of what's left, but that sliver carries outsized business impact. A single hit from GPTBot, PerplexityBot, or ChatGPT-User isn't one user. It's every real user who'll eventually see your content through a large language model instead of on your site. Getting that experience right is worth a little engineering.\n\nThe problem with serving those crawlers HTML: they don't want it. LLM training pipelines and retrieval systems operate on text. So when a crawler pulls your product documentation and needs to turn it into answers, HTML is overhead for them. It has to be parsed, stripped of boilerplate, de-noised of tracking pixels and menu chrome, and flattened into plain text. Some of that cleanup is lossy, especially tables, code blocks, and footnotes, which often show up mangled in downstream summaries.\n\nMarkdown sidesteps most of that, it's what those existing pipelines already speak natively. And it's small, a typical article compresses to 20-30% of its HTML size, which means less bandwidth and fewer tokens burned on your structure instead of your ideas.\n\nThe catch is that rewriting everything to serve Markdown at origin isn't realistic for most teams, and you don't want to anyway. Browsers still need the HTML. What you want is a transform that runs on the request path, doesn't slow things down, and caches well so you're not paying for the same work twice.\n\n## What We're Building\n\nA small JavaScript service on [ Fastly Compute](https://www.fastly.com/products/edge-compute) that sits in front of your origin and does three things based on who's asking:\n\nA normal browser request gets HTML, passed through origin untouched.\n\nAn AI crawler user-agent (we detect 17 of them by default) or a request with\n\n`Accept: text/markdown`\n\ngets a Markdown version of the same page.An explicit\n\n`/md/<path>`\n\nrequest always returns Markdown. Useful for debugging, internal tooling, and content teams who want to spot-check what crawlers see.\n\nHere's what the output looks like for a request to `/md/blog/rate-limits`\n\n:\n\n```\n---\ntitle: \"Rate limits — API docs\"\ndescription: \"How rate limits work, per-tier quotas, and the headers to inspect.\"\nauthor: \"Platform team\"\ndate: \"2026-03-02T00:00:00Z\"\nurl: \"https://example.com/docs/rate-limits\"\nsource: \"https://your-site.edgecompute.app/md/blog/rate-limits\"\n---\n\n# Rate limits\n\nEvery API key is subject to a request budget per minute and per day...\n\n## Quotas by tier\n\n| Tier | Requests / min | Requests / day |\n| --- | --- | --- |\n| Free | 60 | 10,000 |\n| Pro | 600 | 500,000 |\n| Enterprise | Custom | Custom |\n```\n\nClean headings, a real Markdown table, YAML frontmatter a downstream pipeline can parse without heuristics. Nav, footer, related-articles, newsletter prompts, inline scripts, are all stripped away.\n\n## The Stack\n\nFour pieces do all the work:\n\nruns the whole thing as WebAssembly, close to the user. We use the JavaScript SDK (__Fastly Compute__`@fastly/js-compute`\n\n).parses the origin HTML into a DOM. It's a lightweight, standards-adjacent implementation that compiles cleanly to WASM, unlike jsdom, which pulls in a lot of Node-specific machinery.__linkedom__extracts the main content. It's a newer extractor from the Obsidian Web Clipper team, purpose-built for agent-facing Markdown. It handles site-specific quirks (per-site extractors for known publications), standardizes code blocks and footnotes into consistent HTML, and falls back to heuristic scoring when it has to.__Defuddle__walks the extracted DOM and emits Markdown. We add the GFM plugin for tables and strikethrough, plus one small custom rule to handle a linkedom quirk (more on that below).__Turndown__\n\nPlus `fastly:cache`\n\n's SimpleCache for edge caching, no other dependencies.\n\n## The Conversion Pipeline\n\nEverything that turns HTML into Markdown lives in one file, `src/converter.js`\n\n:\n\n``` python\nimport Defuddle from 'defuddle';\nimport { parseHTML } from 'linkedom';\nimport TurndownService from 'turndown';\nimport { gfm } from '@joplin/turndown-plugin-gfm';\n\nconst turndown = new TurndownService({\n  headingStyle: 'atx',\n  codeBlockStyle: 'fenced',\n  bulletListMarker: '-',\n});\nturndown.use(gfm);\n\nexport function htmlToMarkdown(html, sourceUrl) {\n  const { document } = parseHTML(html);\n\n  const result = new Defuddle(document, { url: sourceUrl }).parse();\n  const articleDoc = parseHTML(result?.content || '').document;\n  const markdown = turndown.turndown(articleDoc.documentElement).trim();\n\n  if (!markdown) {\n    throw new Error('Could not extract readable content from page');\n  }\n\n  const frontmatter = buildFrontmatter(result, document, sourceUrl);\n  return `${frontmatter}\\n\\n${markdown}\\n`;\n}\n```\n\nThe pipeline is linear: parse with linkedom, hand the `Document`\n\nto Defuddle, let Defuddle do its extraction and standardization, then re-parse its HTML output through linkedom one more time so Turndown has a real DOM node to walk. That second parse feels redundant, but it matters and we'll get to why in a moment.\n\nThe `buildFrontmatter`\n\nhelper pulls title, description, author, and published date from Defuddle's metadata, falling back to standard `<meta>`\n\ntags when Defuddle doesn't have them. We also emit the canonical URL, so whatever consumes this Markdown can point back to the original page.\n\n### The DOM-node-not-string gotcha\n\nIf you read Defuddle's docs, you'll notice a `markdown: true`\n\noption that looks like it should do everything Turndown does for us. It does in Node, but it doesn't in Compute.\n\nThe reason: Defuddle's built-in Markdown step calls `turndownService.turndown(htmlString)`\n\n. Turndown, given a string, parses it internally by calling `document.implementation.createHTMLDocument`\n\n. The Compute JS runtime is SpiderMonkey with linkedom providing the DOM, and linkedom doesn't expose `document.implementation`\n\n. Turndown throws, Defuddle swallows the throw, and you get a fallback message like \"Partial conversion completed with errors\" with the raw HTML appended.\n\nHanding Turndown a DOM node sidesteps that parser entirely. It walks the tree we give it. That's why the second `parseHTML`\n\ncall is there.\n\n### The Table Rule\n\nOne more linkedom quirk: `HTMLTableElement.rows`\n\nisn't populated. The GFM plugin's table rule checks `node.rows[0]`\n\nto decide whether to convert the table or skip it, and since `rows`\n\nis undefined, every table becomes flattened text.\n\nThe fix is a small custom rule registered after GFM:\n\n``` js\nturndown.addRule('linkedom-table', {\n  filter: (node) => node.nodeName === 'TABLE',\n  replacement: (_content, node) => {\n    const rows = Array.from(node.querySelectorAll('tr'));\n    if (!rows.length) return '';\n    const cells = (tr) =>\n      Array.from(tr.querySelectorAll('th, td')).map((c) =>\n        c.textContent.replace(/\\s+/g, ' ').trim().replace(/\\|/g, '\\\\|'),\n      );\n    const header = cells(rows[0]);\n    const body = rows.slice(1).map(cells);\n    const sep = header.map(() => '---');\n    const fmt = (row) => `| ${row.join(' | ')} |`;\n    return `\\n\\n${[fmt(header), fmt(sep), ...body.map(fmt)].join('\\n')}\\n\\n`;\n  },\n});\n```\n\n`querySelectorAll('tr')`\n\nworks where `.rows`\n\ndoesn't. Since our custom rule is registered last, Turndown picks it over the GFM default. A few extra lines that save any page with a table.\n\n## Routing and content negotiation\n\nThe Compute fetch handler lives in `src/index.js`\n\n. The whole routing layer is about 50 lines:\n\n``` js\nasync function handleRequest(event) {\n  const req = event.request;\n  const url = new URL(req.url);\n\n  if (url.pathname === '/health') return jsonResponse({ status: 'ok' });\n  if (url.pathname === '/__html-2-md__') return landingResponse();\n\n  if (url.pathname.startsWith('/md/') || url.pathname === '/md') {\n    const originPath = url.pathname.replace(/^\\/md/, '') || '/';\n    return await convertAndRespond(req, url, originPath);\n  }\n\n  const ua = req.headers.get('User-Agent') || '';\n  const accept = req.headers.get('Accept') || '';\n\n  if (isAiCrawler(ua) || wantsMarkdown(accept)) {\n    return await convertAndRespond(req, url, url.pathname);\n  }\n\n  return fetch(req, { backend: 'origin' });\n}\n```\n\nFour decision points, in order. Health and debug routes are served locally. A `/md/<path>`\n\nprefix forces Markdown regardless of headers. After that, we look at the request: if it's from a known AI crawler or explicitly asks for Markdown, we convert. Otherwise, a straight pass-through to origin.\n\nThe crawler detection is a small list in `src/agents.js`\n\n, 17 user-agent patterns covering the mainstream ones: GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, GoogleOther, cohere-ai, and so on. It's a case-insensitive substring match. Agents evolve, so treat the list as a starting point and prune or extend based on what actually shows up in your logs.\n\n### Caching\n\nMarkdown conversion takes a few hundred milliseconds on a cold request, most of it in Defuddle's scoring. That's fine for the first crawler hit, painful for the hundredth. SimpleCache turns it into a one-liner:\n\n``` js\nconst cacheKey = `html-2-md:${originUrl.pathname}${originUrl.search}`;\nconst cached = SimpleCache.get(cacheKey);\n\nif (cached) {\n  body = await cached.text();\n} else {\n  body = await fetchAndConvert(originUrl, url);\n  SimpleCache.set(cacheKey, body, CACHE_TTL); // 5 minutes\n}\n```\n\nFive minutes is a reasonable default for most content sites, just tune it to how often you publish. The cache is per-POP, so you'll see a cold conversion per region on first request, then cached responses after.\n\nWe also set `Vary: Accept, User-Agent`\n\non the response. Any downstream caches (yours, the crawler's) will respect the same content negotiation we do.\n\n## Testing Locally\n\nThe converter is a pure function, HTML in, Markdown out. That makes it trivial to test with plain Node, no Compute runtime required:\n\n``` python\nimport { test } from 'node:test';\nimport assert from 'node:assert/strict';\nimport { htmlToMarkdown } from '../src/converter.js';\n\ntest('docs page: preserves tables and nested lists', async () => {\n  const html = await readFile('test/fixtures/docs-page.html', 'utf8');\n  const md = htmlToMarkdown(html, 'https://example.com/docs/rate-limits');\n\n  assert.match(md, /# Rate limits/);\n  assert.match(md, /\\|\\s*Tier\\s*\\|/);  // markdown table header\n  assert.match(md, /\\|\\s*Free\\s*\\|\\s*60\\s*\\|/);\n});\n```\n\nDrop a handful of representative fixtures into `test/fixtures/`\n\n(a blog post, a docs page with tables, a news article with boilerplate), and assert on the properties you care about. Our companion repo ships with three. `npm test`\n\nruns in about 200ms, which means you can iterate on extraction quirks without rebuilding WASM.\n\nFor the full edge pipeline, `fastly compute serve`\n\nboots Viceroy (Fastly's local Compute emulator) on `127.0.0.1:7676`\n\n:\n\n```\ncurl -s \"http://127.0.0.1:7676/\" -H \"Accept: text/markdown\" | head -30\ncurl -s \"http://127.0.0.1:7676/\" -H \"User-Agent: GPTBot/1.0\" | head -30\ncurl -s \"http://127.0.0.1:7676/md/blog/my-post\" | head -30\ncurl -sI \"http://127.0.0.1:7676/\"   # confirm HTML pass-through\n```\n\nPoint `[local_server.backends.origin]`\n\nin `fastly.toml`\n\nat whatever origin you want to proxy, and you've got a working end-to-end loop.\n\n## Deploying\n\nSame two commands as any other Compute service:\n\n```\nnpm run build        # compile to bin/main.wasm\nfastly compute deploy\n```\n\nFirst run prompts you to create a service and configure your production origin backend. After that, you've got a Compute endpoint that'll respond at `<service>.edgecompute.app`\n\n. Point a custom domain at it, or front it with your existing Fastly service as a shielding config, whichever fits your topology.\n\n## What's actually happening on the wire\n\nFor a request from GPTBot to `/blog/my-post`\n\n:\n\nCompute gets the request. User-Agent matches\n\n`GPTBot`\n\n→ route to conversion path.Check SimpleCache for\n\n`html-2-md:/blog/my-post`\n\n. Miss.Fetch HTML from origin (the\n\n`origin`\n\nbackend declared in`fastly.toml`\n\n).Parse with linkedom → run Defuddle → re-parse → Turndown → frontmatter.\n\nStore in SimpleCache with 5-minute TTL. Return.\n\nResponse:\n\n`Content-Type: text/markdown`\n\n;`charset=utf-8`\n\n,`Vary: Accept`\n\n,`User-Agent`\n\n,`X-Markdown-Tokens: <estimate>`\n\n.\n\nFor a regular browser hitting the same URL at the same time, step 2 is skipped entirely. They get HTML straight from origin, same as always.\n\n## Where to Take it From Here\n\nA few directions worth considering once it's running:\n\n**Token counting: **Our heuristic (`length / 4`\n\n) is a rough approximation of GPT-style tokenization. If you care about accurate accounting, swap in a real tokenizer. There are WASM-compatible tiktoken builds that work in Compute.\n\n**Link rewriting: **The current output preserves relative URLs from origin, which means a crawler has to resolve them against the request URL. You can rewrite relative links to absolute inside the Defuddle result before Turndown runs it.\n\n**Per-site extractors: **Defuddle supports custom extractors for sites with unusual structure. If you're proxying a specific publication or docs site, writing a one-off extractor produces much cleaner output than the generic heuristics.\n\n**Streaming: **For very long articles, the current implementation buffers the whole body before emitting the response. Streaming the conversion would reduce TTFB. It's more complex (Defuddle wants the full document to score) but feasible by chunking on section boundaries.\n\n**Rate limiting by agent: **If you want to serve GPTBot but throttle a noisier bot, pair this service with our [ Edge Rate Limiting](https://docs.fastly.com/products/edge-rate-limiting) offering.\n\n## Wrapping up\n\nServing Markdown to AI agents is one of those small efforts that can have an outsized impact. It respects the agent’s workload, but also *your* bandwidth (and ultimately your bottom line). Compute is a good fit for it because the work is close to the request, cacheable, and measured in milliseconds. What you want is a transform that runs on the request path, doesn't slow things down, and caches well so you're not paying for the same work twice.\n\nFeel free to clone the service [ here](https://github.com/fastly/html-2-md). If you build something interesting on top of this (a token counter, a custom extractor, a link rewriter), we'd like to hear about it.", "url": "https://wpnews.pro/news/give-ai-agents-the-markdown-they-actually-want", "canonical_source": "https://www.fastly.com/blog/give-ai-agents-markdown-they-actually-want/", "published_at": "2026-05-28 00:00:00+00:00", "updated_at": "2026-06-24 17:48:49.850818+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "large-language-models", "ai-products"], "entities": ["Fastly", "Fastly Compute", "GPTBot", "PerplexityBot", "ChatGPT-User", "WebAssembly", "JavaScript", "Markdown"], "alternates": {"html": "https://wpnews.pro/news/give-ai-agents-the-markdown-they-actually-want", "markdown": "https://wpnews.pro/news/give-ai-agents-the-markdown-they-actually-want.md", "text": "https://wpnews.pro/news/give-ai-agents-the-markdown-they-actually-want.txt", "jsonld": "https://wpnews.pro/news/give-ai-agents-the-markdown-they-actually-want.jsonld"}}