Two weeks of serving Markdown to agents, straight from the nginx logs

GoodBarber, an app platform, has served Markdown versions of every public page for two weeks via two routes — an `Accept: text/markdown` content-negotiated response and a `.md` URL suffix — generating the Markdown from cached HTML through a Django middleware without duplicating templates or content. The company's engineering lead reports that the conversion is byte-identical for browsers (adding roughly eight microseconds per request) and now also feeds nightly regenerated llms.txt and llms-full.txt files in eleven languages. The writeup focuses on how the two doors are built, how requests were counted from dedicated nginx logs, and what those logs can and cannot prove.

Since early September, every public page of our website has had a Markdown twin, reachable by header or by URL. Two dedicated nginx logs have recorded every request through either door since September 4. Two weeks later I read them, all of them, with the method below. The results, in short, are on our blog https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/ . This is the engineering cut: how the two doors are built, how the count was made, and what the logs can and cannot prove. I co-founded GoodBarber, an app platform, and I run its engineering. The site in question is the one that sells it: marketing pages, a blog, a help center, on eleven language hosts. Pierre-Laurent, our head of backend engineering, counted in August four months of requests to our llms.txt https://dev.to/pierrelaurentmedori/llmstxt-in-the-wild-1321-requests-and-not-one-ai-assistant-came-looking-3205 and found no assistant that came on its own. What stayed with me was his other number: over the same four months, assistants and their crawlers made 1.6 million requests to our regular pages, and Claude Code accounted for about 4,500 of them, half on the help center. The index goes unread while the pages get fetched, so what I wanted to know is what a fetch receives. The site is a Django application behind nginx, with a full-page cache. The constraint was mine: no rewriting of templates, no duplicated content. So the Markdown is produced from the HTML the full-page cache already holds. The converting middleware is the last one in Django's list, which makes it the first to see a response on the way out, before minification and before the CSRF placeholder is substituted; that's what keeps the output identical between a cache hit and a miss, and cacheable by a fingerprint of the HTML. Conversion happens once per distinct HTML, and everything else is a lookup. Two doors open onto it. Send Accept: text/markdown on any public page and, if the q-values make Markdown the preferred type, you get it at the same URL, with Vary: Accept . Or add .md to the path: /pricing.md , /blog/<slug .md , /index.md for the root. The .md route resolves the canonical path and hands the request to the same middleware, so the two doors can never disagree. Every eligible HTML page declares its twin twice, in a <link rel="alternate" type="text/markdown" and in a Link header. The twin carries a YAML front matter title, description, canonical URL, dates taken from the page's JSON-LD , a Link back to the canonical page, and X-Robots-Tag: noindex . An unknown .md path gets a short Markdown 404 document, not a converted error page. Twenty-four back-office and partial routes are excluded and answer 404 under .md . Before it went live, we fetched the same three pages with the feature on and off, as seven kinds of client, and compared the twenty-one responses byte for byte: identical. Browsers get the bytes they got before, for eight microseconds more per request. The same conversion now feeds llms.txt and llms-full.txt, regenerated every night in eleven languages by rendering the pages in-process, with the currency forced per country so that the prices in the Markdown match the prices on the page. None of this is original. Cloudflare https://blog.cloudflare.com/markdown-for-agents/ has done the conversion at the edge since February. Zapier https://zapier.com/pricing.md and Vercel https://vercel.com/docs/agent-resources/markdown-access serve both doors, with front matter and the alternate link. We followed a pattern that already existed, on a site that isn't documentation, and counted. One measurement framed my expectations before I opened the logs. Checkly measured https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/ in February which coding agents send the header: Claude Code, Cursor and OpenCode do; Codex, Gemini CLI, Copilot and Windsurf don't. The .md URL is there for the ones that don't. When a Next.js middleware started detecting GPTBot by user agent to serve it Markdown instead of HTML, John Mueller called the idea stupid, and Search Engine Journal noted https://www.searchenginejournal.com/cloudflares-new-markdown-for-ai-bots-what-you-need-to-know/567339/ that his objection was to user-agent sniffing, not to content negotiation. We never look at the user agent to decide what to send, and the Markdown is produced from the same HTML the cache already holds, keyed by its fingerprint, so it can't drift from the page. The count uses three sources, all converted to UTC the logs are in Paris time : | Source | Window | Lines | Carries | |---|---|---|---| | dedicated nginx log for .md URLs, on the app servers | Sep 4 → Sep 18 | about 82,000 | host, Accept , X-Forwarded-For , X-Real-IP , scheme, response time | | dedicated nginx log for requests whose Accept contains text/markdown | Sep 4 → Sep 18 | about 23,000 | same format | | front-end syslog for all traffic | May → Sep 18 | about 260 million | host and X-Forwarded-For | A request is a log line. A visitor is a triple of IP, user agent and UTC day. Every "who" is a user-agent string checked against the IP ranges its operator publishes, downloaded once and dated because some of them rotate daily: OpenAI's three lists, Perplexity's two, Google's four, Bing, Apple, DuckDuckGo, Common Crawl, Mistral, Ahrefs, Amazon's page, and Meta's AS routes. Baidu, Yandex and PetalBot document reverse DNS instead, so their hits were checked by forward-confirmed reverse lookup. Anthropic publishes nothing: a ClaudeBot request can only be matched against the blocks that carry the rest of ClaudeBot's HTML traffic in the same logs, which is an indication, not a proof, and I treat it as declared. Our own traffic is excluded: our test tools and eight internal addresses, every request from them whatever the user agent. For a negotiated request, "received Markdown" is established by comparing the response size with the .md twin of the same path. The counts below come from the two dedicated logs; the front-end log serves for the period before September 4 and for referers. The .md door: about 80,000 requests, almost all GET, from about 5,800 addresses and 20,000 visitors, on about 26,000 distinct host-and-path pairs, almost all of them served at least once with a 200. No 5xx in two weeks. The header door: about 22,500 requests, two thirds GET and one third HEAD, from about 5,700 addresses and 6,700 visitors. Only about 11,900 were served with a 200. About 10,500 were redirects, and some 8,500 of those were the same thing: a request for a page without its trailing slash, which our router answers with a 301 before the middleware ever runs. ShapBot asks for every page that way, HEAD first, then GET, and collects most of the redirects. The negotiated 404s, about 130, got the Markdown 404 document. | Door | Requests | Served 200 | Who, mostly | |---|---|---|---| | .md URL | about 80,000 | 93.8% | crawlers: 51.0% declared AI crawlers, then Baidu, Bing, Apple | | Accept: text/markdown | about 22,500 | 52.7% | answer engines built for agents, and one coding agent | They don't attract the same clients. The .md link is what corpus crawlers use. Amazonbot, about 13,000 requests, every IP in Amazon's published range, no referer. Meta-ExternalAgent, about 10,000, verified, no referer. GPTBot, about 5,800 verified requests, every one of them with a referer from our own site, almost always the HTML twin, which the same client had fetched in the previous ten minutes in 98.9% of cases. OpenAI's crawlers follow our rel="alternate" . Bing's crawler came too, about 6,800 verified .md requests, probably by the same route, though it sends no referer; 91.7% of them target our internal search pages, which answer any query anyone types, so that is where most of Bing's visit went. The real Googlebot asked for zero .md . Every request that called itself Googlebot came from outside Google's ranges. The header is a different population. The big consumer assistants and their crawlers almost never send it: | Declared agent | Page requests seen | With Accept: text/markdown | |---|---|---| | ShapBot | about 12,000 | 98.3% | | ExaSearchBot | about 4,700 | 93.5% | | Claude Code | about 1,400 | 99.1% | | ClaudeBot | about 24,000 | 0.0% | | ChatGPT-User | about 10,000 | 0.0% | | GPTBot | about 7,700 | 0.1% | | PerplexityBot | about 5,800 | 0.1% | ShapBot and ExaSearchBot, two answer engines built for agents, make 80% of the negotiated requests. Neither publishes IP ranges, so those identities are declared, not verified. The third is the visitor Pierre-Laurent had already counted: about 1,400 requests declaring themselves Claude Code, from more than 500 addresses and 35 versions of the client, sending the header on 99.1% of their page requests and receiving Markdown in 97.7% of the cases we can determine, three quarters of them on the help center. Claude Code runs on a developer's machine, so there is no range to check. The .md link was the fallback for the assistants that don't send the header. Here is what the assistants that fetch a page because a human asked them a question did with it: | Agent declared | HTML pages fetched | .md or negotiated | |---|---|---| | ChatGPT-User | about 10,000 | 1 | | Claude-User hosted | about 2,700 | 0 | | Perplexity-User | about 900 | 0 | Minutes after the twins went live, from an OpenAI address: that one was me, testing through ChatGPT. Verified where a range exists: 89.7% of the ChatGPT-User pages sit in OpenAI's ranges; Anthropic publishes none, so hosted Claude-User can't be checked; none of the Perplexity-User requests fall in Perplexity's four published prefixes, so either the list is stale or the name is borrowed. About fourteen thousand fetches declared as made for a human, one of them on a .md , and that one mine. Those assistants fetch the HTML and, presumably, convert it on their side; in fourteen days, neither the header nor the .md link changed that. What they do with the page once they have it, our logs can't see. Two smaller things the logs said. Nobody arrived on a .md from chatgpt.com, perplexity.ai, claude.ai, a search engine or a social network: zero referers in all those categories. And the Markdown weighs 78.7% less than the HTML page it comes from, median over about 22,000 pages with compression included when the client accepts it, 83.9% on the help center. Bytes on the wire; I didn't count tokens. I went back to May in the front-end log to see what asked for Markdown before we served any. In three and a half months, about 900 requests for .md paths on the site, none of them served: 404s and redirects. README.md , agents.md , SKILL.md , CLAUDE.md , CHANGELOG.md , probed at the root of a marketing site by tools that expect a repository. And in early August, a connector generator guessed the .md URLs of eleven of our pages, the exact pattern we shipped a month later, and got 404s for all of them. llms.txt moved too. Since September 4, OpenAI's crawlers reach it from our own pages, about 25 requests with a page of the site as referer, following the link our pages carry to it; in Pierre-Laurent's window https://dev.to/pierrelaurentmedori/llmstxt-in-the-wild-1321-requests-and-not-one-ai-assistant-came-looking-3205 , the only GPTBot hits with a referer came from a third-party directory. His count continues: since early August, about 1,200 requests for llms.txt, about a hundred from a declared AI operator's crawler or fetcher, nine in ten of them verified, one triggered by a human. Same picture as his four months, at two and a half times the rate: directories, scanners, SEO tools and browsers do the reading, AI operators stay a small minority, and the assistants still don't come for a human: one fetch, against his seventeen. Two access log directives per app server, in the combined format plus the host, the Accept header, X-Forwarded-For , X-Real-IP , the origin scheme and the response time: one file for every request whose Accept contains text/markdown , one for every .md path. Everything else comes from the front-end syslog, which carries the host and X-Forwarded-For . The operators' published IP ranges were downloaded once and dated: one of OpenAI's lists changed on the day I downloaded it, so a verification replayed a month later would not match without that snapshot. The field that would turn the last step, "received Markdown", from a size comparison into a lookup is $sent http content type , next to the status code. The clients to expect are the ones already sending the header, coding agents and the answer engines built for them. The assistants answering humans fetched HTML before the twins existed and still do. If you run content negotiation on a site that isn't documentation: what does your access log record for a negotiated response, and how do you keep your own team's curl out of the count? I'll answer in the comments with our numbers.