Making your site visible to LLMs and AI agents: 6 techniques that work, 8 that don't

Evil Martians reported that more than half of AI agent reads on evilmartians.com in September and October 2026 were Markdown requests, almost all from the coding agent Claude Code via content negotiation, after six months of server-log analysis. The dev agency's updated guide ranks serving Markdown to clients sending `Accept: text/markdown` and keeping HTML clean as the techniques that work, while `llms.txt` — its top pick in April 2026 — is still read by almost no AI tool. Evil Martians said the split matters because coding agents like Claude Code, Cursor and OpenCode request Markdown, while chat assistants including ChatGPT, Perplexity, DeepSeek and DuckDuckGo read the same HTML a browser gets.

Making your site visible to LLMs and AI agents: 6 techniques that work, 8 that don't We recently signed a new client, and they actually found us via Claude. The founders of an SF-based AI startup asked Claude to recommend a dev agency with senior engineers who think about architecture and scale. Evil Martians came back as the top recommendation. That got us thinking: what did we do right? Other parts: 1. Making your site visible to LLMs and AI agents: 6 techniques that work, 8 that don't 2. Which AI actually reads your site? Two months of AI agent and LLM traffic, measured https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured 3. Publish agent skills like packages: discovery index, digests, and install sources https://evilmartians.com/chronicles/publishing-agent-skills-discovery-index In this post: six techniques we deployed to make evilmartians.com https://evilmartians.com/ more discoverable to LLMs and AI agents, and which we actively distribute to our client projects. Updated in October 2026. Since this post came out, we’ve collected six months of our own server logs https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured , and Google has published its guide to generative AI features in Search https://developers.google.com/search/docs/fundamentals/ai-optimization-guide . Both changed the ranking below. The changelog changelog lists every edit. The core idea is simple: LLMs understand clean, well-structured text better than anything else. Everything in this guide is about getting your best content to LLMs in the cleanest format possible. TL;DR: Keep your HTML clean, because ChatGPT and other chat assistants read nothing else. Serve Markdown to clients that send Accept: text/markdown , which is how coding agents like Claude Code ask for it. Add llms.txt last. Almost no AI tool reads it yet, but it’s cheap and could still become a standard. Then measure with server-side logs. Want Claude to do this for you? We packaged the guide as a Claude skill you can download: every technique below, plus the eight anti-patterns to refuse. Save it as SKILL.md inside ~/.claude/skills/llms-visibility/ , then ask Claude to audit your site against it. Encoding the rules in a skill keeps future conversations from re-litigating the unproven techniques. When this post first went out in April 2026, most of these techniques were emerging conventions, not proven standards, and we said so. Six months of logs later, we know more. Some techniques did better than we expected. In September and October, more than half of agent reads on evilmartians.com were Markdown https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured more-than-half-of-agent-reads-are-markdown , and almost all of it went to one coding agent, Claude Code, through content negotiation. Others did worse: llms.txt , April’s top pick, is still read by almost no AI tool. The llms.txt section what-is-llmstxt-and-do-ai-tools-read-it has the numbers. The logs also showed two kinds of AI reader that want different formats: - Coding agents Claude Code, Cursor, OpenCode ask for Markdown and say so in the Accept header https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/ . - Chat assistants ChatGPT, Perplexity, DeepSeek, DuckDuckGo fetch the same HTML a browser gets and read that https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured why-optimize-for-llms-is-the-wrong-instruction . Content negotiation serves the first group, and clean HTML serves the second. Convention-based techniques like llms.txt and hidden hints serve neither: they’re cheap, but mostly unread. The list below is re-ranked around that split, and the engineering cost of all of it is still near zero. Search is no longer the only discovery channel. In 2024, Gartner predicted https://www.gartner.com/en/newsroom/press-releases/2024-02-19-gartner-predicts-search-engine-volume-will-drop-25-percent-by-2026-due-to-ai-chatbots-and-other-virtual-agents a 25% drop in traditional search volume by 2026. The numbers are tracking close: Google searches per US user fell nearly 20% year-over-year https://searchengineland.com/google-searches-per-us-user-fall-report-468051 in 2025, and Google search traffic to publishers dropped 33% https://reutersinstitute.politics.ox.ac.uk/journalism-media-and-technology-trends-and-predictions-2026 between November 2024 and November 2025. When someone asks Perplexity about a topic, pastes a URL into ChatGPT, or has a coding agent like Cursor fetch documentation, they’re relying on whatever the model can extract from your site. That extraction is often messy, lossy, or skipped entirely. Yet, the web already has the machinery to fix that mess. HTTP content negotiation, <link tags, structured Markdown endpoints—none of this is exotic. What is new: LLMs and AI agents have given us concrete reasons to wire it all up. As mentioned, the engineering costs are low and the payoff can be big. Most of the techniques below take under an hour to implement. robots.txt letting AI crawlers in? One thing before the list: check your robots.txt . None of the techniques below matter if you’re blocking AI crawlers from reaching your site in the first place. Many default configurations disallow bots like GPTBot and ClaudeBot. A quick audit takes ten minutes and is the actual step zero. Know which bot each rule hits, though. OpenAI documents three user agents https://developers.openai.com/api/docs/bots : - GPTBot crawls pages for model training. - OAI-SearchBot indexes pages for ChatGPT search. - ChatGPT-User fetches a page when a person asks ChatGPT about it mid-conversation. OpenAI says “robots.txt rules may not apply” to it. So disallowing GPTBot keeps you out of training data, but it has no effect on ChatGPT-User , which is our second-largest AI reader https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured why-optimize-for-llms-is-the-wrong-instruction . Check your CDN’s bot settings too. Cloudflare, for example, can block AI crawlers at the edge https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/ , and that setting is separate from your robots.txt . While you’re in there, consider a Content-Signal: line, an emerging convention from Cloudflare https://contentsignals.org/ released under CC0 that tells crawlers what they’re allowed to do with the bytes after they fetch them. It splits intent into three orthogonal signals: search use in search results , ai-input use as live context for an AI answer, like when Perplexity cites your page or ChatGPT fetches it mid-conversation , and ai-train use to train or fine-tune a model . Here’s the line we ship: User-agent: Content-Signal: search=yes, ai-input=yes, ai-train=yes This is finer-grained than a blanket allow/disallow: a crawler can be welcome to cite you in a live answer without also being cleared to train on you, or any other mix. Set the values to match your own policy. We allow all three in our robots.txt https://evilmartians.com/robots.txt , and the isitagentready.com https://isitagentready.com scanner now treats the line as a baseline check, alongside robots.txt and sitemap presence. If you’re on Cloudflare, you may already ship this line. Its managed robots.txt adds it on over 3.8 million domains https://blog.cloudflare.com/content-signals-policy/ with search=yes, ai-train=no and no ai-input value. Keep in mind that these signals are preferences, not enforcement. A crawler can still ignore them. We set ai-train=yes on purpose, the opposite of Cloudflare’s default. In the conversation that opened this post, Claude didn’t fetch our site. It recommended us from what it already knew. For a consultancy, being in the training data is marketing. A model trained on your site knows which services you offer and what you recommend, and it can repeat both without ever touching your server. No llms.txt file or Markdown route reaches that conversation; only training does. If you sell the content itself, the math is different, so decide which business you’re in before you copy anyone’s line. Worth flagging: Content-Signal: is not part of RFC 9309 https://www.rfc-editor.org/rfc/rfc9309.html , so strict validators like Google’s robots.txt tester flag it as an unknown directive. That’s a warning, not a failure; it’s the same RFC requires crawlers to ignore lines they don’t recognize, so the rest of your robots.txt keeps working. Place the line inside a User-agent: group as Cloudflare’s spec shows, so it scopes to all crawlers; the validator warning is unavoidable until testers update their directive lists. Below are six techniques, in the order you’d build them, and a section on what doesn’t work what-ai-seo-techniques-dont-work . | | Mechanism | Importance | Effort | |---|---|---|---| | 0 | robots.txt audit + Content-Signal: line is-your-robotstxt-letting-ai-crawlers-in | Critical | Low, one-line edit | | 1 | Clean, server-rendered HTML why-does-clean-html-matter-for-llms | Critical | Medium, ongoing | | 2 | .md routes why-are-md-routes-the-thing-that-actually-matters | Critical | Low–Medium, route + content work | | 3 | <link tag + HTTP Link header how-do-llms-discover-your-markdown-version-link-tags-and-http-link-headers | Medium | Low, template + middleware | | 4 | Accept: text/markdown content negotiation what-is-content-negotiation-for-llms | Critical | Low–Medium, server logic | | 5 | /llms.txt what-is-llmstxt-and-do-ai-tools-read-it | Low | Low–Medium, static file + curation | | 6 | /llms-full.txt when-should-you-use-llms-fulltxt | Low | Low, static file or redirect | | + | Schema.org / JSON-LD what-about-schemaorg-and-json-ld | Indirect | Low | | + | Agent skills + .well-known/agent-skills/index.json what-about-agent-skills | High, if you have practices to share | Medium | | + | Server-side analytics on AI endpoints ok-how-do-you-know-any-of-this-is-working | Operational | Medium | Clean HTML matters because every chat assistant in our logs reads HTML and nothing else. ChatGPT-User , the fetcher that pulls your page when someone asks ChatGPT about it mid-conversation, asked for Markdown on under 1% of its requests https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured why-optimize-for-llms-is-the-wrong-instruction to our site. The other chat assistants we see DuckDuckGo, DeepSeek, Qwen, Perplexity’s fetcher, and Google’s behave the same way. Together, they make up over a third of our AI traffic, all reading the same HTML your visitors get. And AI traffic is no rounding error. Even with the training and indexing crawlers excluded, agents made three requests for every four human pageviews https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured how-much-of-your-traffic-is-even-human . Google’s guidance for its generative AI features https://developers.google.com/search/docs/fundamentals/ai-optimization-guide points the same way: AI Overviews and AI Mode are “rooted in our core Search ranking and quality systems,” and there’s no special file or markup to add. In April, this section recommended a visually hidden <div that told AI readers where to find the Markdown. We tagged its URL with ?ref=hint to count follows, and since then the count has been zero https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured what-surprised-us . Ours is still on the page because it costs nothing, but it now sits in the list of what doesn’t work what-ai-seo-techniques-dont-work . Here’s what clean HTML means in practice: - Server-render the content. A fetcher that doesn’t execute your bundle sees whatever is in the HTML response. If that’s an empty <div id="root" , that’s what the model reads. - Use real semantics. Use one <h1 and keep heading levels in order. Put the body in <main or <article , and navigation and footers in their own landmarks. Make links real <a href elements, not click handlers on a <div . Agents use headings and links to navigate a page and decide what to fetch next, and a button they can’t follow is a dead end. The HtmlRAG research https://arxiv.org/abs/2411.02959 found that this structure helps retrieval. - Keep the <head light. Inline stylesheets, hydration payloads, and analytics snippets all come before your first paragraph, and fetchers read on a budget. Claude Code’s WebFetch, for example, reads up to 100,000 characters of a page per call https://code.claude.com/docs/en/tools-reference . We’ve seen agents give up on a heavy page and search for it instead. Then they read a search snippet, or whatever a third party says about you. - Keep tables as tables and code as <pre <code . Both convert cleanly to Markdown. That matters because Claude Code https://code.claude.com/docs/en/tools-reference and Cloudflare’s converter https://blog.cloudflare.com/markdown-for-agents/ turn your HTML into Markdown anyway when you don’t serve it yourself. - Make the structured data match the visible text. Google’s AI features page https://developers.google.com/search/docs/appearance/ai-features asks for this, and it’s where JSON-LD still helps more on that below what-about-schemaorg-and-json-ld . None of this is new advice. It’s the same semantic HTML that accessibility and SEO have asked for all along, and AI assistants are now a large share of its readers. .md routes the thing that actually matters? Everything else on this list is a pointer, and well, this is what those pointers point to. The llmstxt.org specification https://llmstxt.org/ proposes that any page with useful content should provide a clean Markdown version at the same URL with .md appended, so example.com/blog/my-post would have a twin at example.com/blog/my-post.md . For the site root or index pages, that means /index.md . This .md convention is what makes the rest of the ecosystem work. When a user pastes example.com/llms.txt into ChatGPT, the AI follows the links. If those links resolve to clean Markdown rather than HTML soup, the quality of responses jumps. A typical HTML page is 80% navigation, scripts, and boilerplate; the Markdown version is 100% content. On a real-world blog post, that means going from ~15,000 tokens of HTML to ~3,000 tokens of Markdown, an 80% reduction. Cloudflare measured the same ratio on its own announcement post: 16,180 tokens as HTML, 3,150 as Markdown https://blog.cloudflare.com/markdown-for-agents/ . For an LLM working within a limited context window, that difference represents the gap between understanding your page and simply giving up on it. When we first published, the only evidence was negative: a CDN log analysis of two high-traffic sites https://www.longato.ch/llm-md-files/ DA 90, millions of daily visits found zero requests to .md files from GPTBot, ClaudeBot, or PerplexityBot, even when those files were listed in llms.txt . Our own logs tell a different story. More than half of agent reads on evilmartians.com are Markdown https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured more-than-half-of-agent-reads-are-markdown , but the readers aren’t the training crawlers from that study. Nearly all of those Markdown reads come from Claude Code, which gets Markdown through content negotiation what-is-content-negotiation-for-llms . ChatGPT-User , by comparison, took Markdown on under 1% of its requests. So .md routes do get read by a specific set of clients you can measure, and they’re what every other technique on this list delivers. They just don’t replace your HTML. There’s also a counterargument worth hearing: the HtmlRAG research https://arxiv.org/abs/2411.02959 found that HTML retains semantic structure headings, metadata, table layouts that plain text or Markdown can strip away, and that this actually improves retrieval performance in RAG pipelines. Clean, semantic HTML may already be more legible to LLMs than we assume. The strongest case for .md routes isn’t that Markdown is inherently better but that most real-world HTML is so bloated with navigation, scripts, and boilerplate that the signal-to-noise ratio is terrible. If your HTML is already clean and semantic, the gap narrows considerably. One practical concern: maintaining two versions of every page means content can drift. If you update the HTML and forget the Markdown endpoint, LLMs get stale information. Automate the conversion or serve from a single source of truth. // Route handler for /blog/:slug export async function handleRequest request: Request { const url = new URL request.url ; const post = await getPost url.pathname ; // Serve .md at the same path with .md suffix if url.pathname.endsWith '.md' { return new Response post.markdownContent, { headers: { 'Content-Type': 'text/markdown; charset=utf-8' }, } ; } return renderHTML post ; } Vercel’s implementation guide https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation shows how to do this for blog content stored in a CMS as rich text, converting to Markdown on the fly in a route handler. If your content is already authored in Markdown as most developer blogs are , you can serve the source directly. <link tags and HTTP Link headers These are two ways to advertise the same thing a Markdown version of the current page to two different classes of client. <link tag This tag goes in your <head and is readable by any crawler that processes HTML: <link rel="alternate" type="text/markdown" href="https://evilmartians.com/blog/my-post.md" / It’s the same mechanism used for RSS feed discovery type="application/rss+xml" and alternate language versions hreflang . The rel="alternate" link type has been part of the WHATWG HTML specification https://html.spec.whatwg.org/multipage/links.html link-type-alternate since HTML4, and the text/markdown MIME type has been officially registered since RFC 7763 https://datatracker.ietf.org/doc/html/rfc7763 . This isn’t experimental territory. Link header This carries the same semantics, but at the protocol level, so it’s visible before a single byte of body is read. Some AI agents, especially autonomous agents and coding assistants, never parse the HTML body. Instead, they make an HTTP request and read the headers: Link: </blog/my-post.md ; rel="alternate"; type="text/markdown" In a server middleware, one line covers every page: js function addMarkdownLink request: Request, response: Response { const url = new URL request.url ; response.headers.set 'Link', <${url.pathname}.md ; rel="alternate"; type="text/markdown" ; return response; } Deploy both. The HTML tag catches crawlers that process the DOM. The HTTP header catches headless fetchers that don’t. Together, they cover the full range of clients. Accept: text/markdown content negotiation is the standards-based way to serve AI-friendly content. In April, we called it the technique most likely to become the default. Six months later, it’s the one coding agents actually use. Content negotiation serves coding agents the way clean HTML why-does-clean-html-matter-for-llms serves chat assistants. No chat assistant in our logs sends the header, so this technique does nothing for ChatGPT and almost everything for Claude Code. Who sends the header? In February 2026, Checkly tested seven coding agents https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/ by pointing each one at an echo endpoint. Three ask for Markdown: Claude Code, Cursor, and OpenCode. Four don’t: OpenAI Codex, GitHub Copilot, Gemini CLI, and Windsurf. Anthropic documents the behavior https://code.claude.com/docs/en/tools-reference : Claude Code’s WebFetch “sets a User-Agent header beginning with Claude-User , and an Accept header that prefers Markdown over HTML so servers that support content negotiation can return Markdown directly.” In our logs https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured why-optimize-for-llms-is-the-wrong-instruction , Claude Code accounts for 94% of all Markdown traffic. The same logs suggest the survey is already out of date: GitHub Copilot took Markdown on every request it made to us. HTTP content negotiation https://www.rfc-editor.org/rfc/rfc9110 section-12 has been part of the web since HTTP/1.1 1997 —it’s how your browser requests HTML and an API client requests JSON from the same URL. Extending this to Markdown means a client sends: Accept: text/markdown, text/html;q=0.9 …and the server returns Markdown if it can, HTML otherwise. Same URL, same content, different representation. The Vary: Accept response header tells CDNs to cache them separately. Combined with the HTTP Link header the-http-link-header and the .md URL convention why-are-md-routes-the-thing-that-actually-matters , a single handler covers all three: type AcceptEntry = { type: string; q: number }; function parseAccept accept: string : AcceptEntry { if accept.trim return { type: ' / ', q: 1 } ; return accept .split ',' .map s = s.trim .filter Boolean .map item = { const type, ...params = item.split ';' .map s = s.trim ; let q = 1; for const p of params { const eq = p.indexOf '=' ; if eq < 0 || p.slice 0, eq .trim .toLowerCase == 'q' continue; const n = parseFloat p.slice eq + 1 ; if Number.isFinite n q = n; } return { type: type.toLowerCase , q }; } ; } function qualityFor entries: AcceptEntry , candidate: string : number { const primary = candidate.split '/' ; let bestSpec = -1; let bestQ = 0; for const { type, q } of entries { const spec = type === candidate ? 3 : type === ${primary}/ ? 2 : type === ' / ' ? 1 : -1; if spec < 0 continue; if spec bestSpec || spec === bestSpec && q bestQ { bestSpec = spec; bestQ = q; } } return bestQ; } function hasExplicitType entries: AcceptEntry , candidate: string : boolean { return entries.some e = e.type === candidate && e.q 0 ; } // markdownContent and htmlContent are produced elsewhere—by your renderer, CMS, or static build export async function handleRequest request: Request { const url = new URL request.url ; const isMdUrl = url.pathname.endsWith '.md' ; const htmlPath = isMdUrl ? url.pathname.replace /\.md$/, '' : url.pathname; const mdLink = <${htmlPath}.md ; rel="alternate"; type="text/markdown" ; const htmlLink = <${htmlPath} ; rel="alternate"; type="text/html" ; if isMdUrl { return new Response markdownContent, { headers: { 'Content-Type': 'text/markdown; charset=utf-8', 'Vary': 'Accept', 'Link': htmlLink, }, } ; } const accept = parseAccept request.headers.get 'accept' ?? '' ; const mdQ = qualityFor accept, 'text/markdown' ; const htmlQ = qualityFor accept, 'text/html' ; if mdQ === 0 && htmlQ === 0 { return new Response 'Not Acceptable', { status: 406, headers: { 'Content-Type': 'text/plain', 'Vary': 'Accept', 'Link': ${htmlLink}, ${mdLink} , }, } ; } if hasExplicitType accept, 'text/markdown' && mdQ = htmlQ { return new Response markdownContent, { headers: { 'Content-Type': 'text/markdown; charset=utf-8', 'Vary': 'Accept', 'Link': htmlLink, }, } ; } return new Response htmlContent, { headers: { 'Content-Type': 'text/html; charset=utf-8', 'Vary': 'Accept', 'Link': mdLink, }, } ; } Two things in this handler are worth calling out, because both matter for real clients. First, it returns 406 Not Acceptable https://www.rfc-editor.org/rfc/rfc9110 section-15.5.7 when neither format is wanted, and second, it compares q-values https://www.rfc-editor.org/rfc/rfc9110 section-12.4.2 instead of substring-matching Accept . Returning 406 is significant because silent substitution is worse than refusal. If you ship HTML to a client that asked for application/json , the mismatch isn’t caught at the boundary; it’s caught later, when a parser somewhere chokes on a wall of <div tags and the failure surfaces several systems away from where it actually started. A 406 is the clean machine-readable signal that lets the client recover and ask for something else. The q-value comparison matters because Accept headers can rank formats instead of treating them as equally acceptable. When a client uses that ranking, substring-matching gets it backward. A header like Accept: text/html, text/markdown;q=0.5 means “I’ll take Markdown, but I really want HTML.” Read naively, it looks like “client wants Markdown” and you ship the wrong format: a browser visiting your page gets a raw .md blob where it expected a rendered page, and an agent that listed HTML first because it needs the layout gets stripped Markdown instead. Comparing the actual preference scores which is what qualityFor is doing above makes the highest-q format win. Both behaviors are part of the acceptmarkdown.com https://acceptmarkdown.com readiness check, and their importance will continue to increase as agentic clients start treating Accept as a real contract instead of a hint. Content negotiation requires no site-specific knowledge from the client. Any agent that sends the right header gets Markdown automatically, from any conforming server. It’s infrastructure, not convention. If we had to bet on which technique on this list survives in five years, it’s this one. That’s because it doesn’t depend on anyone agreeing on a new file format; it’s just HTTP doing what HTTP was designed to do. Google has flagged serving different content to bots as cloaking https://developers.google.com/search/docs/essentials/spam-policies cloaking . The important distinction: content negotiation at the same URL with the same content in a different format, declared via Vary: Accept , is how HTTP has always worked. This is different from serving bots a different article entirely. The precedent Accept: application/json vs Accept: text/html is 25 years old. Nobody called your REST API cloaked for returning JSON. llms.txt , and do AI tools read it? In April, this was the top of the list. Now it’s near the bottom. Ship it, because it’s cheap, but don’t expect AI tools to read it. llms.txt is a Markdown file at your site root that gives AI systems a curated map of your most important content. Think of it as a robots.txt for the age of LLMs. This was proposed in September 2024 https://www.answer.ai/posts/2024-09-03-llmstxt.html by Jeremy Howard of Answer.AI co-founder of fast.ai and former president of Kaggle . Howard’s rationale: “We need something like robots.txt , but designed to help AI systems understand and use website content effectively.” The format is described at llmstxt.org https://llmstxt.org/ . It’s intentionally minimal: an H1 with the site name, a blockquote summary, and H2 sections of annotated links. My Project A tool for doing X. Built for developers who care about Y. Documentation - Quick Start https://evilmartians.com/docs/start : Get up and running in 5 minutes - API Reference https://evilmartians.com/docs/api : Full endpoint documentation Optional - Changelog https://evilmartians.com/changelog : Recent updates Curate it. The spec makes the distinction itself https://llmstxt.org/ existing-standards : “While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs.” Your sitemap lists every URL. llms.txt lists the pages worth reading, grouped by topic, with a one-line description for each link. Ours https://evilmartians.com/llms.txt starts with the key pages, followed by sections for services, clients, open source projects, posts, talks, and the team. Every link points to the page’s .md version. Several major AI tools can use this file when given the URL. Anthropic publishes their own https://platform.claude.com/llms.txt , and so does OpenAI https://developers.openai.com/llms.txt for its developer docs. The tooling ecosystem keeps growing: Mintlify https://www.mintlify.com/docs/ai/llmstxt and GitBook https://gitbook.com/docs/getting-started/llm-ready-docs auto-generate it natively, while Docusaurus https://github.com/rachfop/docusaurus-plugin-llms and VitePress https://github.com/okineadev/vitepress-plugin-llms support it via community plugins. A community directory https://directory.llmstxt.cloud tracks over 3,800 public implementations, up from about 2,100 when we first published. No major LLM provider has committed to reading this file, and two 2026 log studies show their bots mostly don’t: - Ahrefs checked 137,210 domains https://ahrefs.com/blog/llmstxt-study/ in its web analytics that had traffic in May 2026. About 38,000 had a valid llms.txt , and 97% of those got zero requests for it that month. AI retrieval bots made 1.1% of the requests that did arrive. The most frequent AI fetcher was GPTBot , followed by Claude Code. - A 30-day audit of Adobe Experience Manager domains https://www.longato.ch/llmstxt-2026-june/ counted 22,494 requests to /llms.txt . Only 258 of them 1.1% came from verifiable LLM agents, and Googlebot was the largest named crawler. Not one request had /llms.txt as its referrer, so nothing followed the links inside. Our own logs https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured does-anything-actually-read-llmstxt show the same thing on a smaller scale. In September and October, /llms.txt and /llms-full.txt were fetched 593 times. Named AI assistants made 47 of those requests, and 43 of the 47 came from Claude Code after a person pointed it at our site. ChatGPT fetched the file once. The rest came from SEO crawlers, AI-readiness scanners, llms.txt census scripts, and curl . The file did refer 57 page views, but all of them came from two SEO crawlers mapping the site, not from an AI assistant. The one counterexample is Mintlify’s 2025 CDN analysis https://www.mintlify.com/blog/how-often-do-llms-visit-llms-txt of 25 documentation sites. Over seven days, it found a median of 14 visits to llms.txt and 79 to llms-full.txt , most of the latter from ChatGPT. Developer docs attract different agents than a marketing site or a blog, so your results will depend on your audience. Google Search doesn’t use the file either, and now says so in writing. Its guide to generative AI features https://developers.google.com/search/docs/fundamentals/ai-optimization-guide , updated in July 2026, puts llms.txt on the list of things you can skip: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search including its generative AI capabilities , as Google Search itself doesn’t use them.” Google’s John Mueller said the same on Bluesky https://bsky.app/profile/johnmu.com/post/3lrshm4gggs2v in 2025, less formally: “no AI system currently uses llms.txt.” Tooling is the one thing that has moved, and that came from Google too. Lighthouse 13.3 https://github.com/GoogleChrome/lighthouse/releases/tag/v13.3.0 added an Agentic browsing category to its default report, including an llms.txt audit https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt . The audit only checks the file’s format, and Chrome’s docs call the file “optional at the moment.” So Google Search ignores llms.txt , while Google’s Lighthouse checks it. So should you ship it? Yes, after everything above. It costs nothing to serve, it works as a README when someone points an LLM or a coding agent at the URL, and if the format ever becomes a standard, you’re ready. Just don’t expect many readers. The proposal is two years old https://www.answer.ai/posts/2024-09-03-llmstxt.html , and no AI provider has adopted it yet. Ahrefs https://ahrefs.com/blog/llmstxt-study/ adds two tips worth following: - Link to the file from your pages and docs. Agents fetch it only when something points them to it. - Treat it like code you deploy. Keep it to plain links and descriptions, with nothing that reads like an instruction. A file written for LLMs is also a prompt-injection surface. /llms-full.txt ? This is the companion to /llms.txt . While not formally part of the llmstxt.org specification https://llmstxt.org/ , it’s a widely adopted convention https://www.mintlify.com/blog/what-is-llms-txt popularized by documentation platforms like Mintlify and Fern. It’s also the most debatable item on this list. While /llms.txt is a curated index of links, /llms-full.txt is the full text of your site: everything, in one place, no links to follow. The idea is that an AI tool can ingest an entire documentation set in a single fetch, without crawling individual pages. For smaller sites, this can be your full content concatenated into one Markdown file. For larger sites, it’s more practical to redirect to your primary content index /index.md or serve a representative subset. The scale varies wildly. Cloudflare’s llms-full.txt https://developers.cloudflare.com/llms-full.txt runs to roughly 11 million tokens; Vercel’s VP of Product Lee Robinson described theirs as “a 400,000-word novel.” https://x.com/leerob/status/1930687146790146515 Zod https://zod.dev/llms-full.txt , by contrast, keeps theirs to a focused 250 KB. Whether dumping an entire site into one file is actually useful or just performative thoroughness, we’re honestly not sure. For documentation sites, it makes sense: an LLM can build a complete understanding of your API in a single fetch. For a marketing site or a blog? Probably overkill. A redirect to /index.md is fine. In 2025, llms-full.txt looked like it might matter more than llms.txt . Mintlify’s CDN log analysis https://www.mintlify.com/blog/how-often-do-llms-visit-llms-txt of 25 companies found a median of 79 visits to it against 14 to llms.txt , mostly from ChatGPT. That suggested LLMs would rather read complete documentation than follow links. On our site https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured does-anything-actually-read-llmstxt , it went the other way: 95 fetches of /llms-full.txt against 498 of /llms.txt in September and October. The difference is almost certainly the audience again. If you run developer docs, ship the full version. If you run a blog or a marketing site, the redirect is fine. Everything above is about agents reading your pages. Since July, we’ve also published something for agents to install, and it became a discovery channel of its own. An agent skill https://agentskills.io/ is a SKILL.md file that teaches a coding agent a repeatable practice. We publish ours at /agent-skills https://evilmartians.com/agent-skills and list them in .well-known/agent-skills/index.json , the discovery index from Cloudflare’s open RFC https://github.com/cloudflare/agent-skills-discovery-rfc . Running npx skills add https://evilmartians.com reads that index and installs what it lists. Unlike llms.txt , this well-known file has a consumer that actually reads it, because the installer https://github.com/vercel-labs/skills can’t work without it. Each of our skills opens with a one-line credit, and that line goes into every agent session that loads the skill. Make your skill useful on its own, and keep the credit to one sentence. To measure interest, count clicks on the install buttons, not downloads. In the logs, an install looks the same as the skill directories and scanners that fetch the files every day. Our skill cards have had 1,198 clicks since July. We wrote up how to publish and distribute agent skills https://evilmartians.com/chronicles/publishing-agent-skills-discovery-index in a separate post: This is the section we wish we’d had before we started. We evaluated every “AI SEO” technique we could find across 30+ blog posts, GitHub repos, and spec proposals, then checked our own against six months of logs. Eight have no evidence behind them, and some are actively misleading. One of the eight was ours. - <meta name="ai-content-url" has no specification. No origin, no proposal, no AI tool that reads it. We couldn’t find a single source for where this tag came from, yet it appears in multiple “optimize for AI” blog posts. - <meta name="llms" was proposed via llmsmetatags.org https://llmsmetatags.org/ and submitted to the WHATWG HTML spec as issue 11548 https://github.com/whatwg/html/issues/11548 , which was closed as “not planned.” The only implementation is the proposer’s own website. - /.well-known/ai.txt and /ai.txt have multiple competing proposals, none with meaningful adoption. - HTML comments < -- AI-READABLE-VERSION -- don’t work because most LLM parsers strip HTML comments before processing. ChatGPT, Claude, and Perplexity all work with rendered text, not raw HTML source. No crawler documentation mentions reading comments for discovery. - Human/AI toggle buttons are decorative if you already serve .md routes and content negotiation. AI agents don’t click buttons. - User-Agent sniffing to serve Markdown means detecting AI crawlers by User-Agent string and automatically serving them Markdown instead of HTML. This is cloaking https://developers.google.com/search/docs/essentials/spam-policies cloaking : serving different content based on who the visitor is, not what they asked for. Google penalizes it. The standards-compliant alternative is Accept: text/markdown content negotiation, where the client explicitly requests the format. - Dedicated “AI info pages” are standalone pages labeled “for AI assistants” containing structured company facts. There’s no evidence any crawler or retrieval system treats these differently from any other well-structured page on your site. A good /llms.txt and clean Markdown routes already solve this without a special page. - Hidden “for AI” hints in the page were one of the six techniques in the first version of this post: a visually hidden <div pointing to the Markdown URL, tagged so we could count follows. After six months of logs, the count is zero https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured what-surprised-us . Agents that want Markdown ask for it with an Accept header, and the rest just read the page. If you already have a hint, leave it, but don’t build a new one. All of these are solutions looking for a problem that standard web mechanisms already solve. The pattern is always the same: someone invents a new file or meta tag, writes a blog post about it, and other blog posts cite that blog post as evidence. Nobody checks whether any AI system actually reads the thing. We fell for it too, and we only found out because we measured. This foundational GEO study https://arxiv.org/abs/2311.09735 from Princeton and IIT Delhi published at KDD 2024, one of the first rigorous studies of generative engine optimization backs this up. The researchers tested nine content optimization strategies across 10,000 queries. Their findings: including direct quotations increased AI visibility by ~43%, adding statistics boosted it by ~33%, and citing authoritative sources improved it by 115% for previously low-ranked content. Every strategy that worked was about enriching the visible text that LLMs actually read, not about metadata they don’t. Yes, this paragraph citing an authoritative source with specific statistics is itself a GEO technique. You’re welcome, future LLMs. Structured data doesn’t help LLMs read your page, but it does feed the search indexes that many AI answers are built on. In April, this post listed it among the techniques that don’t work. That was only half right. The part that holds is that models still don’t read JSON-LD off the page: - In the SearchVIU experiment https://www.searchviu.com/en/schema-markup-and-ai-in-2025-what-chatgpt-claude-perplexity-gemini-really-see/ , ChatGPT, Claude, Perplexity, Gemini, and Google AI Mode all missed product data placed only in JSON-LD. - OtterlyAI repeated the test https://otterly.ai/blog/schema-markup-real-impact-ai-search/ across seven platforms and 319 prompts. Only Gemini could read a page’s JSON-LD. - When the data does get through, it’s read as plain text. In Mark Williams-Cook’s test https://www.seroundtable.com/chatgpt-perplexity-structured-data-text-40862.html , ChatGPT and Perplexity returned an address that existed only inside invalid JSON-LD. - Ahrefs tracked 1,885 pages https://ahrefs.com/blog/schema-ai-citations/ that added JSON-LD against 4,000 controls and found no citation gain on any platform. The part we got wrong is that many answer engines don’t read your page directly. They read a search index, and the index does read your structured data. Google’s AI Overviews and AI Mode are rooted in its core Search systems https://developers.google.com/search/docs/fundamentals/ai-optimization-guide , which still use markup for rich results and ask that it “matches the visible text on the page” https://developers.google.com/search/docs/appearance/ai-features . Merchant Center reads schema.org/Product https://support.google.com/merchants/answer/6069143 to keep prices and availability current, and Bing pairs the same markup with IndexNow https://blogs.bing.com/webmaster/2025/5/IndexNow-Enables-Faster-and-More-Reliable-Updates-for-Shopping-and-Ads/ to update “search results, shopping experiences, and AI-driven assistants.” Commerce agents skip the markup entirely. OpenAI’s product feed https://developers.openai.com/commerce/specs/file-upload/products is a CSV, TSV, or JSONL file, and the Universal Commerce Protocol https://ucp.dev/ runs on REST and JSON-RPC with built-in MCP support. Neither mentions Schema.org. So our verdict changed from “doesn’t work” to “works indirectly.” Keep the JSON-LD you have, keep it consistent with the visible page, and if you sell things, get into the feeds. Don’t add schema for the model’s benefit, and don’t rip it out because the model can’t see it. Here’s the uncomfortable truth: you don’t, unless you measure. Cloudflare’s 2025 Radar Year in Review https://blog.cloudflare.com/radar-2025-year-in-review/ ai-user-action-crawling-increased-by-over-15x-in-2025 shows AI user-action crawling bots fetching pages to answer real user queries increased over 15x in 2025, with GPTBot, ClaudeBot, and PerplexityBot among the most active crawlers. But raw crawler traffic isn’t the same as visibility. You need to know which of your endpoints are actually being fetched and by whom. Instrument your Markdown endpoints, /llms.txt , and /llms-full.txt with whatever analytics you already use. Track by User-Agent to separate AI crawler traffic from human requests, and by referrer hostname to catch traffic arriving from chatgpt.com , claude.ai , and perplexity.ai . This isn’t cloaking. You’re logging who fetches what, not changing what you serve. js const url = new URL request.url ; if url.pathname.endsWith '.md' { const ua = request.headers.get 'user-agent' ?? ''; const ref = request.headers.get 'referer' ?? ''; analytics.track 'markdown fetch', { ua, ref, path: url.pathname } ; } Since no major LLM provider has made formal commitments to any of these conventions, empirical measurement is the only reliable signal. Traditional client-side analytics won’t help here because AI crawlers don’t execute JavaScript. You need server-side request logs that capture raw User-Agent headers. Since April, two first-party reports have appeared that need none of this instrumentation. Google Search Console’s Generative AI performance report https://developers.google.com/search/docs/fundamentals/ai-optimization-guide shows how your pages perform in AI Overviews and AI Mode. Bing Webmaster Tools has an AI Performance report https://blogs.bing.com/webmaster/2026/2/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview/ for Copilot, in public preview. Both cover only search-based answer engines. The coding agents and on-demand fetchers that make up most of our agent traffic don’t show up in either, so you still need server-side logs. We learned two lessons from running ours for six months https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured : - Classify by exclusion. Don’t rely on a list of known AI user agents, because new ones show up every week. Rule out browsers, search crawlers, and scanners, and count what’s left. - Split each provider’s crawler from its on-demand fetcher. Our single Claude bucket hid the fact that nearly all of it was the Claude Code coding agent, not the training crawler. And in September and October https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured how-much-of-your-traffic-is-even-human , crawlers made more than twice as many requests as the clients fetching pages for a person. Two public scanners turn this from “I think it’s working” into a passing or failing checklist; both are worth running before considering your work here to be done. acceptmarkdown.com https://acceptmarkdown.com is the narrow one: it tests your content negotiation against four specific criteria. Does the URL serve Markdown for Accept: text/markdown ? Does it set Vary: Accept ? Does it return 406 for unsupported types? Does it honor q-values? It also publishes server recipes for Nginx, Caddy, Cloudflare Workers, Next.js, Astro, Rails, Express, Django, and others, which save a lot of guessing if you’re wiring this up for the first time. Here’s what a passing scan looks like on our own site: isitagentready.com https://isitagentready.com is the broader scan. It checks the visibility surface this post focuses on robots.txt , sitemap, Link headers, Markdown negotiation , but also flags a second tier of agent-readiness conventions that go beyond pure content visibility: Content Signals https://contentsignals.org/ in robots.txt , Web Bot Auth https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/ HTTP message signatures so legitimate AI crawlers can prove who they are , MCP server cards https://modelcontextprotocol.io/ and Agent Skills https://agentskills.io/ for sites that want to expose tools to agents, and emerging agentic-commerce protocols like x402 https://www.x402.org/ . You can see our own scan as a Content Site https://isitagentready.com/evilmartians.com?profile=content : Most of those last items are more about what an agent can do on your site than what it can read , so they’re outside the scope of this post. But the line between “site for humans that LLMs read” and “site for agents that humans also read” is going to keep moving, and the scanner is a useful map of where it’s headed. If you ship one thing today, ship content negotiation with .md routes behind it, then re-run both scanners until they’re green. The fixes they surface tend to be one-line changes a missing Vary header, a wrong status code, an absent Content-Signal: line rather than architectural rework. These techniques stack because different AI systems enter your site through different paths. ChatGPT-User reads your rendered HTML. Claude Code sends Accept: text/markdown and takes whatever you negotiate. OAI-SearchBot fetches .md URLs directly. A developer configuring Cursor with your docs pastes /llms-full.txt . Each path needs its own technique. If you’re starting from zero: 1. Audit your robots.txt and add a Content-Signal: line. Make sure you’re not blocking the crawlers you want, know which of OpenAI’s three bots each rule affects, and tell the ones you allow what they can do with your content. 2. Make the HTML clean and server-rendered. Every chat assistant in our logs reads nothing else. 3. Serve .md routes for every page. The payload everything below delivers. 4. Add <link rel="alternate" and the HTTP Link header. One template change plus one middleware line, on both representations. 5. Implement Accept: text/markdown content negotiation, with q-values, a 406 for unsupported types, and Vary: Accept . This is what delivers most of the Markdown. 6. Instrument server-side. Classify by exclusion, and split each provider’s crawler from its on-demand fetcher. 7. Add /llms.txt . Keep expectations low, and link to it from your pages. 8. Run acceptmarkdown.com https://acceptmarkdown.com and isitagentready.com https://isitagentready.com to confirm the wiring before you move on. The goal isn’t gaming AI systems; rather, we want to make your content legible to them, for the same reason you’ve been writing semantic HTML for years. The audience now includes language models and AI agents, so the techniques are different, but the principles are the same. …and frankly, most of this is just good web hygiene that we should’ve been doing anyway. Start with content negotiation and clean HTML. Together, they cover both kinds of AI reader in our logs: coding agents that ask for Markdown, and chat assistants that only read HTML. Content negotiation needs .md routes behind it anyway, and the rest can come later. No. rel="alternate" links, registered MIME types, content negotiation with Vary: Accept —this is all standard HTTP. We’ve been running it on evilmartians.com https://evilmartians.com/ since February 2026 without any ranking changes. Google’s own guidance https://developers.google.com/search/docs/fundamentals/ai-optimization-guide says Search doesn’t use llms.txt or Markdown files, not that it penalizes them. /llms.txt automatically? No. In September and October, our llms.txt and llms-full.txt got 593 requests, and only 47 came from named AI assistants, 43 of them Claude Code. Larger 2026 studies by Ahrefs https://ahrefs.com/blog/llmstxt-study/ and Longato https://www.longato.ch/llmstxt-2026-june/ traced about 1.1% of requests for the file to AI bots. The llms.txt section what-is-llmstxt-and-do-ai-tools-read-it has the details. Ship the file anyway, after content negotiation. Accept: text/markdown ? As of February 2026, Claude Code, Cursor, and OpenCode https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/ do; OpenAI Codex, GitHub Copilot, Gemini CLI, and Windsurf don’t. Cloudflare’s Markdown for Agents https://blog.cloudflare.com/markdown-for-agents/ names Claude Code and OpenCode as the agents it already sees sending the header. Expect the list to grow, and check your own logs for the current one. Not for that reason. Controlled tests https://otterly.ai/blog/schema-markup-real-impact-ai-search/ show that most models still can’t read JSON-LD on the page. Keep it for the search indexes and shopping systems that AI answers are built on, and keep it consistent with the visible text. If you sell products, get into the product feeds https://developers.openai.com/commerce/specs/file-upload/products too. See the section above what-about-schemaorg-and-json-ld . See the tracking section above ok-how-do-you-know-any-of-this-is-working , and the post where we did it https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured . Short version: log on the server, classify by exclusion, and split each provider’s crawler from its on-demand fetcher. Yes. The vocabulary hasn’t settled yet. Agent-friendly, agent-ready, AI-readable, LLM SEO, and GEO generative engine optimization all mean the same thing: a site that AI agents and chat assistants can fetch and cite. Whichever name you use, the work is the six techniques above, and the proof is the same server log. - Re-rated every technique after six months of our own server logs https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured : content negotiation and clean HTML joined .md routes at Critical, <link tags and Link headers dropped to Medium, llms.txt and llms-full.txt to Low, and the sections now run in build order, without their old numbers. - Reordered the rollout to match: clean HTML is now step two, and llms.txt , formerly step two, is step seven. The table gained rows for JSON-LD and agent skills. - Replaced the hidden <div hint zero attributable fetches with clean, server-rendered HTML; the hint moved to the list of what doesn’t work. - Moved Schema.org/JSON-LD from “doesn’t work” to “works indirectly.” - Added agent skills as a discovery channel; OpenAI’s three user agents, CDN bot settings, and why we allow training to step zero; Search Console’s and Bing’s AI reports to the measurement section; and FAQ answers on which coding agents send Accept: text/markdown and on JSON-LD. - Added the 2026 sources: Google’s guide to generative AI features https://developers.google.com/search/docs/fundamentals/ai-optimization-guide , Cloudflare’s Markdown for Agents https://blog.cloudflare.com/markdown-for-agents/ , the Claude Code docs, Lighthouse’s llms.txt audit, and the llms.txt log studies from Ahrefs and Longato. - Original post published. Curious what this looks like in practice? Append .md to any URL on evilmartians.com https://evilmartians.com/ , or check our /llms.txt https://evilmartians.com/llms.txt .