` . Agents use headings and links to navigate a page and decide what to fetch next, and a button they can’t follow is a dead end. The[HtmlRAG research](https://arxiv.org/abs/2411.02959) found that this structure helps retrieval.\n- **Keep the `` light.** Inline stylesheets, hydration payloads, and analytics snippets all come before your first paragraph, and fetchers read on a budget. Claude Code’s WebFetch, for example,[reads up to 100,000 characters of a page per call](https://code.claude.com/docs/en/tools-reference) . We’ve seen agents give up on a heavy page and search for it instead. Then they read a search snippet, or whatever a third party says about you.\n- **Keep tables as tables and code as `
`.** Both convert cleanly to Markdown. That matters because[Claude Code](https://code.claude.com/docs/en/tools-reference) and[Cloudflare’s converter](https://blog.cloudflare.com/markdown-for-agents/) turn your HTML into Markdown anyway when you don’t serve it yourself.\n- **Make the structured data match the visible text.** Google’s[AI features page](https://developers.google.com/search/docs/appearance/ai-features) asks for this, and it’s where JSON-LD still helps ([more on that below](#what-about-schemaorg-and-json-ld) ).\n\nNone of this is new advice. It’s the same semantic HTML that accessibility and SEO have asked for all along, and AI assistants are now a large share of its readers.\n\n## `.md` routes the thing that actually matters?\n\nEverything else on this list is a pointer, and well, this is what those pointers point to.\n\nThe [llmstxt.org specification](https://llmstxt.org/) proposes that any page with useful content should provide a clean Markdown version at the same URL with `.md` appended, so `example.com/blog/my-post` would have a twin at `example.com/blog/my-post.md`. For the site root or index pages, that means `/index.md`.\n\nThis `.md` convention is what makes the rest of the ecosystem work. When a user pastes `example.com/llms.txt` into ChatGPT, the AI follows the links. If those links resolve to clean Markdown rather than HTML soup, the quality of responses jumps. A typical HTML page is 80% navigation, scripts, and boilerplate; the Markdown version is 100% content. On a real-world blog post, that means going from ~15,000 tokens of HTML to ~3,000 tokens of Markdown, an 80% reduction. Cloudflare measured the same ratio on its own announcement post: [16,180 tokens as HTML, 3,150 as Markdown](https://blog.cloudflare.com/markdown-for-agents/).\n\nFor an LLM working within a limited context window, that difference represents the gap between understanding your page and simply giving up on it.\n\n### \n\nWhen we first published, the only evidence was negative: a [CDN log analysis of two high-traffic sites](https://www.longato.ch/llm-md-files/) (DA > 90, millions of daily visits) found zero requests to `.md` files from GPTBot, ClaudeBot, or PerplexityBot, even when those files were listed in `llms.txt`.\n\nOur own logs tell a different story. [More than half of agent reads on evilmartians.com are Markdown](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured#more-than-half-of-agent-reads-are-markdown), but the readers aren’t the training crawlers from that study. Nearly all of those Markdown reads come from Claude Code, which gets Markdown through [content negotiation](#what-is-content-negotiation-for-llms). `ChatGPT-User`, by comparison, took Markdown on under 1% of its requests. So `.md` routes do get read by a specific set of clients you can measure, and they’re what every other technique on this list delivers. They just don’t replace your HTML.\n\nThere’s also a counterargument worth hearing: the [HtmlRAG research](https://arxiv.org/abs/2411.02959) found that HTML retains semantic structure (headings, metadata, table layouts) that plain text or Markdown can strip away, and that this actually *improves* retrieval performance in RAG pipelines. Clean, semantic HTML may already be more legible to LLMs than we assume.\n\nThe strongest case for `.md` routes isn’t that Markdown is inherently better but that most real-world HTML is so bloated with navigation, scripts, and boilerplate that the signal-to-noise ratio is terrible. If your HTML is already clean and semantic, the gap narrows considerably.\n\nOne practical concern: maintaining two versions of every page means content can drift. If you update the HTML and forget the Markdown endpoint, LLMs get stale information. Automate the conversion or serve from a single source of truth.\n\n```\n// Route handler for /blog/:slug\nexport async function handleRequest(request: Request) {\n const url = new URL(request.url);\n const post = await getPost(url.pathname);\n\n // Serve .md at the same path with .md suffix\n if (url.pathname.endsWith('.md')) {\n return new Response(post.markdownContent, {\n headers: { 'Content-Type': 'text/markdown; charset=utf-8' },\n });\n }\n\n return renderHTML(post);\n}\n```\n\n[Vercel’s implementation guide](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation) shows how to do this for blog content stored in a CMS as rich text, converting to Markdown on the fly in a route handler. If your content is already authored in Markdown (as most developer blogs are), you can serve the source directly.\n\n## `` tags and HTTP `Link` headers\n\nThese are two ways to advertise the same thing (a Markdown version of the current page) to two different classes of client.\n\n### `` tag\n\nThis tag goes in your `` and is readable by any crawler that processes HTML:\n\n```\n\n```\n\nIt’s the same mechanism used for RSS feed discovery (`type=\"application/rss+xml\"`) and alternate language versions (` hreflang`). The `rel=\"alternate\"` link type has been part of the [WHATWG HTML specification](https://html.spec.whatwg.org/multipage/links.html#link-type-alternate) since HTML4, and the `text/markdown` MIME type has been [officially registered since RFC 7763](https://datatracker.ietf.org/doc/html/rfc7763). This isn’t experimental territory.\n\n### `Link` header\n\nThis carries the same semantics, but at the protocol level, so it’s visible before a single byte of body is read. Some AI agents, especially autonomous agents and coding assistants, never parse the HTML body. Instead, they make an HTTP request and read the headers:\n\n```\nLink: ; rel=\"alternate\"; type=\"text/markdown\"\n```\n\nIn a server middleware, one line covers every page:\n\n``` js\nfunction addMarkdownLink(request: Request, response: Response) {\n const url = new URL(request.url);\n response.headers.set(\n 'Link',\n `<${url.pathname}.md>; rel=\"alternate\"; type=\"text/markdown\"`\n );\n return response;\n}\n```\n\nDeploy both. The HTML tag catches crawlers that process the DOM. The HTTP header catches headless fetchers that don’t. Together, they cover the full range of clients.\n\n## \n\n`Accept: text/markdown` content negotiation is the standards-based way to serve AI-friendly content. In April, we called it the technique most likely to become the default. Six months later, it’s the one coding agents actually use. Content negotiation serves coding agents the way [clean HTML](#why-does-clean-html-matter-for-llms) serves chat assistants. No chat assistant in our logs sends the header, so this technique does nothing for ChatGPT and almost everything for Claude Code.\n\nWho sends the header? In February 2026, [Checkly tested seven coding agents](https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/) by pointing each one at an echo endpoint. Three ask for Markdown: Claude Code, Cursor, and OpenCode. Four don’t: OpenAI Codex, GitHub Copilot, Gemini CLI, and Windsurf. Anthropic [documents the behavior](https://code.claude.com/docs/en/tools-reference): Claude Code’s WebFetch “sets a `User-Agent` header beginning with `Claude-User`, and an `Accept` header that prefers Markdown over HTML so servers that support content negotiation can return Markdown directly.” In [our logs](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured#why-optimize-for-llms-is-the-wrong-instruction), Claude Code accounts for 94% of all Markdown traffic. The same logs suggest the survey is already out of date: GitHub Copilot took Markdown on every request it made to us.\n\n[HTTP content negotiation](https://www.rfc-editor.org/rfc/rfc9110#section-12) has been part of the web since HTTP/1.1 (1997)—it’s how your browser requests HTML and an API client requests JSON from the same URL. Extending this to Markdown means a client sends:\n\n```\nAccept: text/markdown, text/html;q=0.9\n```\n\n…and the server returns Markdown if it can, HTML otherwise. Same URL, same content, different representation. The `Vary: Accept` response header tells CDNs to cache them separately.\n\nCombined with the [HTTP `Link` header](#the-http-link-header) and the [`.md` URL convention](#why-are-md-routes-the-thing-that-actually-matters), a single handler covers all three:\n\n```\ntype AcceptEntry = { type: string; q: number };\n\nfunction parseAccept(accept: string): AcceptEntry[] {\n if (!accept.trim()) return [{ type: '*/*', q: 1 }];\n return accept\n .split(',')\n .map((s) => s.trim())\n .filter(Boolean)\n .map((item) => {\n const [type, ...params] = item.split(';').map((s) => s.trim());\n let q = 1;\n for (const p of params) {\n const eq = p.indexOf('=');\n if (eq < 0 || p.slice(0, eq).trim().toLowerCase() !== 'q') continue;\n const n = parseFloat(p.slice(eq + 1));\n if (Number.isFinite(n)) q = n;\n }\n return { type: type.toLowerCase(), q };\n });\n}\n\nfunction qualityFor(entries: AcceptEntry[], candidate: string): number {\n const [primary] = candidate.split('/');\n let bestSpec = -1;\n let bestQ = 0;\n for (const { type, q } of entries) {\n const spec =\n type === candidate ? 3 :\n type === `${primary}/*` ? 2 :\n type === '*/*' ? 1 : -1;\n if (spec < 0) continue;\n if (spec > bestSpec || (spec === bestSpec && q > bestQ)) {\n bestSpec = spec;\n bestQ = q;\n }\n }\n return bestQ;\n}\n\nfunction hasExplicitType(entries: AcceptEntry[], candidate: string): boolean {\n return entries.some((e) => e.type === candidate && e.q > 0);\n}\n\n// `markdownContent` and `htmlContent` are produced elsewhere—by your renderer, CMS, or static build\nexport async function handleRequest(request: Request) {\n const url = new URL(request.url);\n const isMdUrl = url.pathname.endsWith('.md');\n const htmlPath = isMdUrl ? url.pathname.replace(/\\.md$/, '') : url.pathname;\n const mdLink = `<${htmlPath}.md>; rel=\"alternate\"; type=\"text/markdown\"`;\n const htmlLink = `<${htmlPath}>; rel=\"alternate\"; type=\"text/html\"`;\n\n if (isMdUrl) {\n return new Response(markdownContent, {\n headers: {\n 'Content-Type': 'text/markdown; charset=utf-8',\n 'Vary': 'Accept',\n 'Link': htmlLink,\n },\n });\n }\n\n const accept = parseAccept(request.headers.get('accept') ?? '');\n const mdQ = qualityFor(accept, 'text/markdown');\n const htmlQ = qualityFor(accept, 'text/html');\n\n if (mdQ === 0 && htmlQ === 0) {\n return new Response('Not Acceptable', {\n status: 406,\n headers: {\n 'Content-Type': 'text/plain',\n 'Vary': 'Accept',\n 'Link': `${htmlLink}, ${mdLink}`,\n },\n });\n }\n\n if (hasExplicitType(accept, 'text/markdown') && mdQ >= htmlQ) {\n return new Response(markdownContent, {\n headers: {\n 'Content-Type': 'text/markdown; charset=utf-8',\n 'Vary': 'Accept',\n 'Link': htmlLink,\n },\n });\n }\n\n return new Response(htmlContent, {\n headers: {\n 'Content-Type': 'text/html; charset=utf-8',\n 'Vary': 'Accept',\n 'Link': mdLink,\n },\n });\n}\n```\n\nTwo things in this handler are worth calling out, because both matter for real clients.\n\nFirst, it returns [`406 Not Acceptable`](https://www.rfc-editor.org/rfc/rfc9110#section-15.5.7) when neither format is wanted, and second, it compares [q-values](https://www.rfc-editor.org/rfc/rfc9110#section-12.4.2) instead of substring-matching `Accept`.\n\nReturning 406 is significant because silent substitution is worse than refusal. If you ship HTML to a client that asked for `application/json`, the mismatch isn’t caught at the boundary; it’s caught later, when a parser somewhere chokes on a wall of `` tags and the failure surfaces several systems away from where it actually started. A 406 is the clean machine-readable signal that lets the client recover and ask for something else.\n\nThe q-value comparison matters because `Accept` headers can rank formats instead of treating them as equally acceptable. When a client uses that ranking, substring-matching gets it backward. A header like `Accept: text/html, text/markdown;q=0.5` means “I’ll take Markdown, but I really want HTML.” Read naively, it looks like “client wants Markdown” and you ship the wrong format: a browser visiting your page gets a raw `.md` blob where it expected a rendered page, and an agent that listed HTML first because it needs the layout gets stripped Markdown instead. Comparing the actual preference scores (which is what `qualityFor` is doing above) makes the highest-q format win.\n\nBoth behaviors are part of the [`acceptmarkdown.com`](https://acceptmarkdown.com) readiness check, and their importance will continue to increase as agentic clients start treating `Accept` as a real contract instead of a hint.\n\n### \n\nContent negotiation requires no site-specific knowledge from the client. Any agent that sends the right header gets Markdown automatically, from any conforming server. It’s infrastructure, not convention. If we had to bet on which technique on this list survives in five years, it’s this one. That’s because it doesn’t depend on anyone agreeing on a new file format; it’s just HTTP doing what HTTP was designed to do.\n\n### \n\nGoogle has flagged serving different content to bots as [cloaking](https://developers.google.com/search/docs/essentials/spam-policies#cloaking). The important distinction: content negotiation at the same URL with the same content in a different format, declared via `Vary: Accept`, is how HTTP has always worked. This is different from serving bots a different article entirely. The precedent (`Accept: application/json` vs `Accept: text/html`) is 25 years old. Nobody called your REST API cloaked for returning JSON.\n\n## `llms.txt`, and do AI tools read it?\n\nIn April, this was the top of the list. Now it’s near the bottom. Ship it, because it’s cheap, but don’t expect AI tools to read it.\n\n`llms.txt` is a Markdown file at your site root that gives AI systems a curated map of your most important content. Think of it as a `robots.txt` for the age of LLMs.\n\nThis was [proposed in September 2024](https://www.answer.ai/posts/2024-09-03-llmstxt.html) by Jeremy Howard of Answer.AI (co-founder of fast.ai and former president of Kaggle). Howard’s rationale: “We need something like `robots.txt`, but designed to help AI systems understand and use website content effectively.” The format is described at [llmstxt.org](https://llmstxt.org/). It’s intentionally minimal: an H1 with the site name, a blockquote summary, and H2 sections of annotated links.\n\n```\n# My Project\n\n> A tool for doing X. Built for developers who care about Y.\n\n## Documentation\n\n- [Quick Start](https://evilmartians.com/docs/start): Get up and running in 5 minutes\n- [API Reference](https://evilmartians.com/docs/api): Full endpoint documentation\n\n## Optional\n\n- [Changelog](https://evilmartians.com/changelog): Recent updates\n```\n\nCurate it. The spec [makes the distinction itself](https://llmstxt.org/#existing-standards): “While sitemaps list all pages for search engines, `llms.txt` offers a curated overview for LLMs.” Your sitemap lists every URL. `llms.txt` lists the pages worth reading, grouped by topic, with a one-line description for each link. [Ours](https://evilmartians.com/llms.txt) starts with the key pages, followed by sections for services, clients, open source projects, posts, talks, and the team. Every link points to the page’s `.md` version.\n\nSeveral major AI tools can use this file when given the URL. [Anthropic publishes their own](https://platform.claude.com/llms.txt), and [so does OpenAI](https://developers.openai.com/llms.txt) for its developer docs. The tooling ecosystem keeps growing: [Mintlify](https://www.mintlify.com/docs/ai/llmstxt) and [GitBook](https://gitbook.com/docs/getting-started/llm-ready-docs) auto-generate it natively, while [Docusaurus](https://github.com/rachfop/docusaurus-plugin-llms) and [VitePress](https://github.com/okineadev/vitepress-plugin-llms) support it via community plugins. A [community directory](https://directory.llmstxt.cloud) tracks over 3,800 public implementations, up from about 2,100 when we first published.\n\n### \n\nNo major LLM provider has committed to reading this file, and two 2026 log studies show their bots mostly don’t:\n\n- **[Ahrefs checked 137,210 domains](https://ahrefs.com/blog/llmstxt-study/)** in its web analytics that had traffic in May 2026. About 38,000 had a valid`llms.txt` , and 97% of those got zero requests for it that month. AI retrieval bots made 1.1% of the requests that did arrive. The most frequent AI fetcher was`GPTBot` , followed by Claude Code.\n- **[A 30-day audit of Adobe Experience Manager domains](https://www.longato.ch/llmstxt-2026-june/)** counted 22,494 requests to`/llms.txt` . Only 258 of them (1.1%) came from verifiable LLM agents, and Googlebot was the largest named crawler. Not one request had`/llms.txt` as its referrer, so nothing followed the links inside.\n\n[Our own logs](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured#does-anything-actually-read-llmstxt) show the same thing on a smaller scale. In September and October, `/llms.txt` and `/llms-full.txt` were fetched 593 times. Named AI assistants made 47 of those requests, and 43 of the 47 came from Claude Code after a person pointed it at our site. ChatGPT fetched the file once. The rest came from SEO crawlers, AI-readiness scanners, `llms.txt` census scripts, and `curl`. The file did refer 57 page views, but all of them came from two SEO crawlers mapping the site, not from an AI assistant.\n\nThe one counterexample is [Mintlify’s 2025 CDN analysis](https://www.mintlify.com/blog/how-often-do-llms-visit-llms-txt) of 25 documentation sites. Over seven days, it found a median of 14 visits to `llms.txt` and 79 to `llms-full.txt`, most of the latter from ChatGPT. Developer docs attract different agents than a marketing site or a blog, so your results will depend on your audience.\n\nGoogle Search doesn’t use the file either, and now says so in writing. Its [guide to generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), updated in July 2026, puts `llms.txt` on the list of things you can skip: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” Google’s John Mueller [said the same on Bluesky](https://bsky.app/profile/johnmu.com/post/3lrshm4gggs2v) in 2025, less formally: “no AI system currently uses llms.txt.”\n\nTooling is the one thing that has moved, and that came from Google too. [Lighthouse 13.3](https://github.com/GoogleChrome/lighthouse/releases/tag/v13.3.0) added an Agentic browsing category to its default report, including an [`llms.txt` audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt). The audit only checks the file’s format, and Chrome’s docs call the file “optional at the moment.” So Google Search ignores `llms.txt`, while Google’s Lighthouse checks it.\n\nSo should you ship it? Yes, after everything above. It costs nothing to serve, it works as a README when someone points an LLM or a coding agent at the URL, and if the format ever becomes a standard, you’re ready. Just don’t expect many readers. The proposal is [two years old](https://www.answer.ai/posts/2024-09-03-llmstxt.html), and no AI provider has adopted it yet.\n\n[Ahrefs](https://ahrefs.com/blog/llmstxt-study/) adds two tips worth following:\n\n- **Link to the file** from your pages and docs. Agents fetch it only when something points them to it.\n- **Treat it like code you deploy.** Keep it to plain links and descriptions, with nothing that reads like an instruction. A file written for LLMs is also a prompt-injection surface.\n\n## `/llms-full.txt`?\n\nThis is the companion to `/llms.txt`. While not formally part of the [llmstxt.org specification](https://llmstxt.org/), it’s a [widely adopted convention](https://www.mintlify.com/blog/what-is-llms-txt) popularized by documentation platforms like Mintlify and Fern. It’s also the most debatable item on this list.\n\nWhile `/llms.txt` is a curated index of links, `/llms-full.txt` is the full text of your site: everything, in one place, no links to follow. The idea is that an AI tool can ingest an entire documentation set in a single fetch, without crawling individual pages.\n\nFor smaller sites, this can be your full content concatenated into one Markdown file. For larger sites, it’s more practical to redirect to your primary content index (`/index.md`) or serve a representative subset.\n\nThe scale varies wildly. [Cloudflare’s `llms-full.txt`](https://developers.cloudflare.com/llms-full.txt) runs to roughly 11 million tokens; Vercel’s VP of Product Lee Robinson [described theirs as “a 400,000-word novel.”](https://x.com/leerob/status/1930687146790146515) [Zod](https://zod.dev/llms-full.txt), by contrast, keeps theirs to a focused 250 KB. Whether dumping an entire site into one file is actually useful or just performative thoroughness, we’re honestly not sure. For documentation sites, it makes sense: an LLM can build a complete understanding of your API in a single fetch. For a marketing site or a blog? Probably overkill. A redirect to `/index.md` is fine.\n\nIn 2025, `llms-full.txt` looked like it might matter more than `llms.txt`. [Mintlify’s CDN log analysis](https://www.mintlify.com/blog/how-often-do-llms-visit-llms-txt) of 25 companies found a median of 79 visits to it against 14 to `llms.txt`, mostly from ChatGPT. That suggested LLMs would rather read complete documentation than follow links. [On our site](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured#does-anything-actually-read-llmstxt), it went the other way: 95 fetches of `/llms-full.txt` against 498 of `/llms.txt` in September and October. The difference is almost certainly the audience again. If you run developer docs, ship the full version. If you run a blog or a marketing site, the redirect is fine.\n\n## \n\nEverything above is about agents reading your pages. Since July, we’ve also published something for agents to install, and it became a discovery channel of its own.\n\nAn [agent skill](https://agentskills.io/) is a `SKILL.md` file that teaches a coding agent a repeatable practice. We publish ours at [/agent-skills](https://evilmartians.com/agent-skills) and list them in `.well-known/agent-skills/index.json`, the discovery index from [Cloudflare’s open RFC](https://github.com/cloudflare/agent-skills-discovery-rfc). Running `npx skills add https://evilmartians.com` reads that index and installs what it lists. Unlike `llms.txt`, this well-known file has a consumer that actually reads it, because the [installer](https://github.com/vercel-labs/skills) can’t work without it.\n\nEach of our skills opens with a one-line credit, and that line goes into every agent session that loads the skill. Make your skill useful on its own, and keep the credit to one sentence. To measure interest, count clicks on the install buttons, not downloads. In the logs, an install looks the same as the skill directories and scanners that fetch the files every day. Our skill cards have had 1,198 clicks since July. We wrote up [how to publish and distribute agent skills](https://evilmartians.com/chronicles/publishing-agent-skills-discovery-index) in a separate post:\n\n## \n\nThis is the section we wish we’d had before we started. We evaluated every “AI SEO” technique we could find across 30+ blog posts, GitHub repos, and spec proposals, then checked our own against six months of logs. Eight have no evidence behind them, and some are actively misleading. One of the eight was ours.\n\n- **``** has no specification. No origin, no proposal, no AI tool that reads it. We couldn’t find a single source for where this tag came from, yet it appears in multiple “optimize for AI” blog posts.\n- **``** was[proposed via llmsmetatags.org](https://llmsmetatags.org/) and submitted to the WHATWG HTML spec as[issue #11548](https://github.com/whatwg/html/issues/11548) , which was closed as “not planned.” The only implementation is the proposer’s own website.\n- **`/.well-known/ai.txt` and `/ai.txt`** have multiple competing proposals, none with meaningful adoption.\n- **HTML comments (``)** don’t work because most LLM parsers strip HTML comments before processing. ChatGPT, Claude, and Perplexity all work with rendered text, not raw HTML source. No crawler documentation mentions reading comments for discovery.\n- **Human/AI toggle buttons** are decorative if you already serve`.md` routes and content negotiation. AI agents don’t click buttons.\n- **User-Agent sniffing to serve Markdown** means detecting AI crawlers by User-Agent string and automatically serving them Markdown instead of HTML. This is[cloaking](https://developers.google.com/search/docs/essentials/spam-policies#cloaking) : serving different content based on who the visitor is, not what they asked for. Google penalizes it. The standards-compliant alternative is`Accept: text/markdown` content negotiation, where the client explicitly requests the format.\n- **Dedicated “AI info pages”** are standalone pages labeled “for AI assistants” containing structured company facts. There’s no evidence any crawler or retrieval system treats these differently from any other well-structured page on your site. A good`/llms.txt` and clean Markdown routes already solve this without a special page.\n- **Hidden “for AI” hints in the page** were one of the six techniques in the first version of this post: a visually hidden`` pointing to the Markdown URL, tagged so we could count follows. After six months of logs,[the count is zero](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured#what-surprised-us) . Agents that want Markdown ask for it with an`Accept` header, and the rest just read the page. If you already have a hint, leave it, but don’t build a new one.\n\nAll of these are solutions looking for a problem that standard web mechanisms already solve. The pattern is always the same: someone invents a new file or meta tag, writes a blog post about it, and other blog posts cite that blog post as evidence. Nobody checks whether any AI system actually reads the thing. We fell for it too, and we only found out because we measured.\n\nThis [foundational GEO study](https://arxiv.org/abs/2311.09735) from Princeton and IIT Delhi (published at KDD 2024, one of the first rigorous studies of generative engine optimization) backs this up. The researchers tested nine content optimization strategies across 10,000 queries. Their findings: including direct quotations increased AI visibility by ~43%, adding statistics boosted it by ~33%, and citing authoritative sources improved it by 115% for previously low-ranked content. Every strategy that worked was about enriching the visible text that LLMs actually read, not about metadata they don’t. *Yes, this paragraph citing an authoritative source with specific statistics is itself a GEO technique. You’re welcome, future LLMs.*\n\n### \n\nStructured data doesn’t help LLMs read your page, but it does feed the search indexes that many AI answers are built on. In April, this post listed it among the techniques that don’t work. That was only half right.\n\nThe part that holds is that models still don’t read JSON-LD off the page:\n\n- In the [SearchVIU experiment](https://www.searchviu.com/en/schema-markup-and-ai-in-2025-what-chatgpt-claude-perplexity-gemini-really-see/) , ChatGPT, Claude, Perplexity, Gemini, and Google AI Mode all missed product data placed only in JSON-LD.\n- [OtterlyAI repeated the test](https://otterly.ai/blog/schema-markup-real-impact-ai-search/) across seven platforms and 319 prompts. Only Gemini could read a page’s JSON-LD.\n- When the data does get through, it’s read as plain text. In Mark Williams-Cook’s [test](https://www.seroundtable.com/chatgpt-perplexity-structured-data-text-40862.html) , ChatGPT and Perplexity returned an address that existed only inside invalid JSON-LD.\n- [Ahrefs tracked 1,885 pages](https://ahrefs.com/blog/schema-ai-citations/) that added JSON-LD against 4,000 controls and found no citation gain on any platform.\n\nThe part we got wrong is that many answer engines don’t read your page directly. They read a search index, and the index does read your structured data. Google’s AI Overviews and AI Mode [are rooted in its core Search systems](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), which still use markup for rich results and ask that it [“matches the visible text on the page”](https://developers.google.com/search/docs/appearance/ai-features). [Merchant Center reads `schema.org/Product`](https://support.google.com/merchants/answer/6069143) to keep prices and availability current, and Bing [pairs the same markup with IndexNow](https://blogs.bing.com/webmaster/2025/5/IndexNow-Enables-Faster-and-More-Reliable-Updates-for-Shopping-and-Ads/) to update “search results, shopping experiences, and AI-driven assistants.”\n\nCommerce agents skip the markup entirely. OpenAI’s [product feed](https://developers.openai.com/commerce/specs/file-upload/products) is a CSV, TSV, or JSONL file, and the [Universal Commerce Protocol](https://ucp.dev/) runs on REST and JSON-RPC with built-in MCP support. Neither mentions Schema.org.\n\nSo our verdict changed from “doesn’t work” to “works indirectly.” Keep the JSON-LD you have, keep it consistent with the visible page, and if you sell things, get into the feeds. Don’t add schema for the model’s benefit, and don’t rip it out because the model can’t see it.\n\n## \n\nHere’s the uncomfortable truth: you don’t, unless you measure. [Cloudflare’s 2025 Radar Year in Review](https://blog.cloudflare.com/radar-2025-year-in-review/#ai-user-action-crawling-increased-by-over-15x-in-2025) shows AI user-action crawling (bots fetching pages to answer real user queries) increased over 15x in 2025, with GPTBot, ClaudeBot, and PerplexityBot among the most active crawlers. But raw crawler traffic isn’t the same as visibility. You need to know which of your endpoints are actually being fetched and by whom.\n\nInstrument your Markdown endpoints, `/llms.txt`, and `/llms-full.txt` with whatever analytics you already use. Track by User-Agent to separate AI crawler traffic from human requests, and by referrer hostname to catch traffic arriving from `chatgpt.com`, `claude.ai`, and `perplexity.ai`. This isn’t cloaking. You’re logging who fetches what, not changing what you serve.\n\n``` js\nconst url = new URL(request.url);\nif (url.pathname.endsWith('.md')) {\n const ua = request.headers.get('user-agent') ?? '';\n const ref = request.headers.get('referer') ?? '';\n analytics.track('markdown_fetch', { ua, ref, path: url.pathname });\n}\n```\n\nSince no major LLM provider has made formal commitments to any of these conventions, empirical measurement is the only reliable signal. Traditional client-side analytics won’t help here because AI crawlers don’t execute JavaScript. You need server-side request logs that capture raw User-Agent headers.\n\nSince April, two first-party reports have appeared that need none of this instrumentation. Google Search Console’s [Generative AI performance report](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) shows how your pages perform in AI Overviews and AI Mode. Bing Webmaster Tools has an [AI Performance report](https://blogs.bing.com/webmaster/2026/2/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview/) for Copilot, in public preview. Both cover only search-based answer engines. The coding agents and on-demand fetchers that make up most of our agent traffic don’t show up in either, so you still need server-side logs.\n\nWe learned two lessons from [running ours for six months](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured):\n\n- **Classify by exclusion.** Don’t rely on a list of known AI user agents, because new ones show up every week. Rule out browsers, search crawlers, and scanners, and count what’s left.\n- **Split each provider’s crawler from its on-demand fetcher.** Our single`Claude` bucket hid the fact that nearly all of it was the Claude Code coding agent, not the training crawler. And in[September and October](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured#how-much-of-your-traffic-is-even-human) , crawlers made more than twice as many requests as the clients fetching pages for a person.\n\n## \n\nTwo public scanners turn this from “I think it’s working” into a passing or failing checklist; both are worth running before considering your work here to be done.\n\n[**`acceptmarkdown.com`**](https://acceptmarkdown.com) is the narrow one: it tests your content negotiation against four specific criteria. Does the URL serve Markdown for `Accept: text/markdown`? Does it set `Vary: Accept`? Does it return `406` for unsupported types? Does it honor q-values? It also publishes server recipes for Nginx, Caddy, Cloudflare Workers, Next.js, Astro, Rails, Express, Django, and others, which save a lot of guessing if you’re wiring this up for the first time. Here’s what a passing scan looks like on our own site:\n\n[**`isitagentready.com`**](https://isitagentready.com) is the broader scan. It checks the visibility surface this post focuses on (`robots.txt`, sitemap, `Link` headers, Markdown negotiation), but also flags a second tier of agent-readiness conventions that go beyond pure content visibility: [Content Signals](https://contentsignals.org/) in `robots.txt`, [Web Bot Auth](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/) (HTTP message signatures so legitimate AI crawlers can prove who they are), [MCP server cards](https://modelcontextprotocol.io/) and [Agent Skills](https://agentskills.io/) for sites that want to expose tools to agents, and emerging agentic-commerce protocols like [x402](https://www.x402.org/). You can see [our own scan as a Content Site](https://isitagentready.com/evilmartians.com?profile=content):\n\nMost of those last items are more about what an agent can *do* on your site than what it can *read*, so they’re outside the scope of this post. But the line between “site for humans that LLMs read” and “site for agents that humans also read” is going to keep moving, and the scanner is a useful map of where it’s headed.\n\nIf you ship one thing today, ship content negotiation with `.md` routes behind it, then re-run both scanners until they’re green. The fixes they surface tend to be one-line changes (a missing `Vary` header, a wrong status code, an absent `Content-Signal:` line) rather than architectural rework.\n\n## \n\nThese techniques stack because different AI systems enter your site through different paths. `ChatGPT-User` reads your rendered HTML. Claude Code sends `Accept: text/markdown` and takes whatever you negotiate. `OAI-SearchBot` fetches `.md` URLs directly. A developer configuring Cursor with your docs pastes `/llms-full.txt`. Each path needs its own technique.\n\nIf you’re starting from zero:\n\n1. **Audit your `robots.txt` and add a `Content-Signal:` line.** Make sure you’re not blocking the crawlers you want, know which of OpenAI’s three bots each rule affects, and tell the ones you allow what they can do with your content.\n2. **Make the HTML clean and server-rendered.** Every chat assistant in our logs reads nothing else.\n3. **Serve `.md` routes for every page.** The payload everything below delivers.\n4. **Add `` and the HTTP `Link` header.** One template change plus one middleware line, on both representations.\n5. **Implement `Accept: text/markdown` content negotiation,** with q-values, a`406` for unsupported types, and`Vary: Accept` . This is what delivers most of the Markdown.\n6. **Instrument server-side.** Classify by exclusion, and split each provider’s crawler from its on-demand fetcher.\n7. **Add `/llms.txt`.** Keep expectations low, and link to it from your pages.\n8. **Run [`acceptmarkdown.com`](https://acceptmarkdown.com) and [` isitagentready.com`](https://isitagentready.com)** to confirm the wiring before you move on.\n\nThe goal isn’t gaming AI systems; rather, we want to make your content legible to them, for the same reason you’ve been writing semantic HTML for years. The audience now includes language models and AI agents, so the techniques are different, but the principles are the same.\n\n…and frankly, most of this is just good web hygiene that we should’ve been doing anyway.\n\n## \n\n### \n\nStart with content negotiation and clean HTML. Together, they cover both kinds of AI reader in our logs: coding agents that ask for Markdown, and chat assistants that only read HTML. Content negotiation needs `.md` routes behind it anyway, and the rest can come later.\n\n### \n\nNo. `rel=\"alternate\"` links, registered MIME types, content negotiation with `Vary: Accept`—this is all standard HTTP. We’ve been running it on [evilmartians.com](https://evilmartians.com/) since February 2026 without any ranking changes. Google’s [own guidance](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says Search doesn’t use `llms.txt` or Markdown files, not that it penalizes them.\n\n### `/llms.txt` automatically?\n\nNo. In September and October, our `llms.txt` and `llms-full.txt` got 593 requests, and only 47 came from named AI assistants, 43 of them Claude Code. Larger 2026 studies by [Ahrefs](https://ahrefs.com/blog/llmstxt-study/) and [Longato](https://www.longato.ch/llmstxt-2026-june/) traced about 1.1% of requests for the file to AI bots. [The `llms.txt` section](#what-is-llmstxt-and-do-ai-tools-read-it) has the details. Ship the file anyway, after content negotiation.\n\n### `Accept: text/markdown`?\n\nAs of February 2026, [Claude Code, Cursor, and OpenCode](https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/) do; OpenAI Codex, GitHub Copilot, Gemini CLI, and Windsurf don’t. Cloudflare’s [Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) names Claude Code and OpenCode as the agents it already sees sending the header. Expect the list to grow, and check your own logs for the current one.\n\n### \n\nNot for that reason. [Controlled tests](https://otterly.ai/blog/schema-markup-real-impact-ai-search/) show that most models still can’t read JSON-LD on the page. Keep it for the search indexes and shopping systems that AI answers are built on, and keep it consistent with the visible text. If you sell products, get into the [product feeds](https://developers.openai.com/commerce/specs/file-upload/products) too. See [the section above](#what-about-schemaorg-and-json-ld).\n\n### \n\nSee the [tracking section above](#ok-how-do-you-know-any-of-this-is-working), and [the post where we did it](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured). Short version: log on the server, classify by exclusion, and split each provider’s crawler from its on-demand fetcher.\n\n### \n\nYes. The vocabulary hasn’t settled yet. Agent-friendly, agent-ready, AI-readable, LLM SEO, and GEO (generative engine optimization) all mean the same thing: a site that AI agents and chat assistants can fetch and cite. Whichever name you use, the work is the six techniques above, and the proof is the same server log.\n\n## \n\n### \n\n- Re-rated every technique after [six months of our own server logs](https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured) : content negotiation and clean HTML joined`.md` routes at Critical,`` tags and`Link` headers dropped to Medium,`llms.txt` and`llms-full.txt` to Low, and the sections now run in build order, without their old numbers.\n- Reordered the rollout to match: clean HTML is now step two, and `llms.txt` , formerly step two, is step seven. The table gained rows for JSON-LD and agent skills.\n- Replaced the hidden `` hint (zero attributable fetches) with clean, server-rendered HTML; the hint moved to the list of what doesn’t work.\n- Moved Schema.org/JSON-LD from “doesn’t work” to “works indirectly.”\n- Added agent skills as a discovery channel; OpenAI’s three user agents, CDN bot settings, and why we allow training to step zero; Search Console’s and Bing’s AI reports to the measurement section; and FAQ answers on which coding agents send `Accept: text/markdown` and on JSON-LD.\n- Added the 2026 sources: Google’s [guide to generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) , Cloudflare’s[Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) , the Claude Code docs, Lighthouse’s`llms.txt` audit, and the`llms.txt` log studies from Ahrefs and Longato.\n\n### \n\n- Original post published.\n\n*Curious what this looks like in practice? Append `.md` to any URL on [evilmartians.com](https://evilmartians.com/), or check our [`/llms.txt`](https://evilmartians.com/llms.txt).*", "url": "https://wpnews.pro/news/making-your-site-visible-to-llms-and-ai-agents-6-techniques-that-work-8-that-don", "canonical_source": "https://evilmartians.com/chronicles/how-to-make-your-website-visible-to-llms", "published_at": "2026-11-04 00:00:00+00:00", "updated_at": "2026-11-04 18:54:13.067263+00:00", "lang": "en", "topics": ["generative-engine-optimization", "ai-crawlers", "ai-agents", "ai-search", "agent-protocols"], "entities": ["Evil Martians", "Claude Code", "ChatGPT", "Perplexity", "DeepSeek", "DuckDuckGo", "Cursor", "OpenCode"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/making-your-site-visible-to-llms-and-ai-agents-6-techniques-that-work-8-that-don", "markdown": "https://wpnews.pro/news/making-your-site-visible-to-llms-and-ai-agents-6-techniques-that-work-8-that-don.md", "text": "https://wpnews.pro/news/making-your-site-visible-to-llms-and-ai-agents-6-techniques-that-work-8-that-don.txt", "jsonld": "https://wpnews.pro/news/making-your-site-visible-to-llms-and-ai-agents-6-techniques-that-work-8-that-don.jsonld"}}