llms.txt Explained: What It Is, Whether AI Answer Engines Read It, and How to Write One

A developer at Merlonix explains the llms.txt convention, a proposed standard for a Markdown file at a domain root that curates key content for large language models. The post clarifies that llms.txt is not an access-control file like robots.txt but a table of contents, and addresses the current state of adoption and consumption by AI answer engines.

Originally published on the Merlonix blog. /llms.txt is the file everyone adding "AI SEO" to their site keeps hearing about and nobody quite explains. It gets lumped in with robots.txt because it lives in the same place and has a similar name, so people assume it's another access-control file — a way to let the good AI crawlers in or keep them out. It is not. robots.txt says who may fetch what . llms.txt says here is what matters, and where it is . One is a gate; the other is a table of contents. Adding an llms.txt grants no access and blocks nothing — a crawler that a WAF or robots.txt rule is keeping out will never see it, and a crawler that's already allowed in doesn't need your permission to read it. What it does is make a site legible to an agent that has to otherwise reverse-engineer your navigation to find the three pages that actually answer the question. Here's the whole thing: what the file is, the exact format, what to put in it, the honest answer to "does anything actually read it yet," and how to confirm yours is valid rather than just present. llms.txt is a proposed convention — published at llmstxt.org https://llmstxt.org/ in late 2024 — for a single Markdown file at your domain root https://example.com/llms.txt that curates your most important content for a large language model. The premise is simple: an LLM working with a limited context window can't crawl and hold your entire site, and rendered marketing HTML is noisy — nav, cookie banners, footers, scripts. A short, hand-curated Markdown file that names your key pages and links straight to them is far cheaper for an agent to consume than making it guess which of your 400 URLs are the canonical docs. It is not a standard in the RFC sense — it's a community convention with growing adoption, mostly among documentation sites and developer tools. It is also not : robots.txt llms.txt is curation and grants nothing. sitemap.xml is an exhaustive, machine-generated list of llms.txt is the opposite: short, hand-picked, human-written, and llms.txt says "here are the six things that matter." this is an Article , this is a Product . llms.txt operates at the site level, pointing across pages.Think of it as the README you'd hand a new hire who has ten minutes: not the whole codebase, just "start here, then here, and here's where pricing lives." The spec is deliberately minimal, and it's just Markdown, so it renders fine for a human too. The shape: Project or Site Name An optional one-line blockquote summary of what this site or project is, so an agent gets the gist immediately. Some optional free-form Markdown here — a paragraph of context, key facts, whatever an LLM should know first. Docs - Getting started https://example.com/docs/start : the five-minute setup - API reference https://example.com/docs/api : every endpoint, typed Guides - Deploying to production https://example.com/guides/deploy - Troubleshooting TLS https://example.com/guides/tls Optional - Changelog https://example.com/changelog The pieces that matter: immediately after, holding a short summary. Agents are told to treat this as the elevator pitch. : a short description of what's behind it. Optional sectionThere's also a companion file, llms-full.txt , for sites that want to inline the llms.txt links; llms-full.txt includes. Most sites only need the former.The failure mode isn't a syntax error — it's an llms.txt that's technically valid and practically useless because it lists your blog's forty most recent posts, or duplicates your nav. Curate it the way you'd brief a smart stranger: Pricing … : four tiers from free to $699/mo, no per-seat billing is a citable fact; a bare link isn't.This is the question that matters and the one most "add an llms.txt today " posts skip, so here it is straight: adoption on the publishing side is real and growing; consumption by the major answer engines is not something anyone can currently guarantee. OpenAI, Anthropic, Google, and Perplexity have not committed to reading llms.txt at retrieval time the way they commit to honoring robots.txt . Some developer-facing tools and agent frameworks do look for it. The big consumer answer engines mostly still crawl and parse your rendered pages directly. So why add one? Three reasons that hold regardless of whether ChatGPT reads it this quarter: What you should not do is treat it as a substitute for the things engines demonstrably do use today — being crawlable robots.txt + no WAF rule silently 403-ing the answer engines and being parseable JSON-LD structured data . llms.txt is the third layer, not the first. If an answer engine can't fetch your pages at all, a perfect llms.txt changes nothing. Getting cited by an AI answer engine is three questions in order, and llms.txt is only the last one: robots.txt llms.txt — the curated table of contents.Skip step 1 and the other two are moot. Nail all three and you've done everything a site can currently do to be legible to an AI agent. "Present" and "valid" are different. A file that returns 200 but is empty, or is a wall of prose with no headings or links, is present-but-useless — and an agent parsing it as Markdown gets nothing structured out of it. Verify it the way a consumer would: curl -s -o /dev/null -w "%{http code}\n" https://example.com/llms.txt A 404 means you don't have one. A 200 that serves your HTML error page a SPA catch-all route is worse — it "exists" but is garbage. llms.txt has at least one heading or one Markdown link text url . If it has neither, a parser can't extract structure from it.If you'd rather not eyeball it, the free AI Agent-Readiness checker https://merlonix.com/tools/agent-readiness/ does exactly this from outside your stack: it fetches your /llms.txt , checks that it's present and shaped like real Markdown a heading or a link — not an empty file or an error page served in its place , and folds that into a 0–100 score alongside the two things that matter more today — whether your robots.txt actually lets the answer-engine crawlers in, and whether your pages carry JSON-LD structured data. It tells you which of the three layers is your weak point, which is the only question worth acting on. No signup, one domain at a time. The one-line version: llms.txt is a curated, Markdown table of contents you write for AI agents — cheap to add, standards-track, and legible, but it is not access control and it is not yet something the major answer engines promise to read. Add it because it's a near-free option on a likely future and a canonical statement of your key pages — but add it Merlonix watches all three layers the way it watches SSL, DNS, and domain expiry: continuously, and from outside your infrastructure, so a redeploy that drops your llms.txt , a new WAF rule that 403s the answer engines, or a template change that strips your JSON-LD doesn't quietly erode your AI-answer visibility weeks before anyone notices the referral traffic fall off. Run the free agent-readiness scan https://merlonix.com/tools/agent-readiness/ to see where a domain stands today, check its live SSL and DNS https://merlonix.com/tools/domain-health/ while you're there, and browse the rest of the free tools https://merlonix.com/tools/ . Being findable by an agent is a configuration problem you can mostly fix this afternoon — once you know which of the three layers is the one holding you back.