{"slug": "llms-txt-explained-what-it-is-whether-ai-answer-engines-read-it-and-how-to-write", "title": "llms.txt Explained: What It Is, Whether AI Answer Engines Read It, and How to Write One", "summary": "A developer at Merlonix explains the llms.txt convention, a proposed standard for a Markdown file at a domain root that curates key content for large language models. The post clarifies that llms.txt is not an access-control file like robots.txt but a table of contents, and addresses the current state of adoption and consumption by AI answer engines.", "body_md": "*Originally published on the Merlonix blog.*\n\n`/llms.txt`\n\nis the file everyone adding \"AI SEO\" to their site keeps hearing about and nobody quite explains. It gets lumped in with `robots.txt`\n\nbecause it lives in the same place and has a similar name, so people assume it's another access-control file — a way to let the good AI crawlers in or keep them out. It is not. `robots.txt`\n\nsays *who may fetch what*. `llms.txt`\n\nsays *here is what matters, and where it is*. One is a gate; the other is a table of contents. Adding an `llms.txt`\n\ngrants no access and blocks nothing — a crawler that a WAF or robots.txt rule is keeping out will never see it, and a crawler that's already allowed in doesn't *need* your permission to read it. What it does is make a site legible to an agent that has to otherwise reverse-engineer your navigation to find the three pages that actually answer the question.\n\nHere's the whole thing: what the file is, the exact format, what to put in it, the honest answer to \"does anything actually read it yet,\" and how to confirm yours is valid rather than just present.\n\n`llms.txt`\n\nis a proposed convention — published at [llmstxt.org](https://llmstxt.org/) in late 2024 — for a single Markdown file at your domain root (`https://example.com/llms.txt`\n\n) that curates your most important content for a large language model. The premise is simple: an LLM working with a limited context window can't crawl and hold your entire site, and rendered marketing HTML is noisy — nav, cookie banners, footers, scripts. A short, hand-curated Markdown file that names your key pages and links straight to them is far cheaper for an agent to consume than making it guess which of your 400 URLs are the canonical docs.\n\nIt is **not** a standard in the RFC sense — it's a community convention with growing adoption, mostly among documentation sites and developer tools. It is also **not**:\n\n`robots.txt`\n\n`llms.txt`\n\nis curation and grants nothing.`sitemap.xml`\n\nis an exhaustive, machine-generated list of `llms.txt`\n\nis the opposite: short, hand-picked, human-written, and `llms.txt`\n\nsays \"here are the six things that matter.\"`this is an Article`\n\n, `this is a Product`\n\n). `llms.txt`\n\noperates at the site level, pointing across pages.Think of it as the README you'd hand a new hire who has ten minutes: not the whole codebase, just \"start here, then here, and here's where pricing lives.\"\n\nThe spec is deliberately minimal, and it's just Markdown, so it renders fine for a human too. The shape:\n\n```\n# Project or Site Name\n\n> An optional one-line blockquote summary of what this\n> site or project is, so an agent gets the gist immediately.\n\nSome optional free-form Markdown here — a paragraph of\ncontext, key facts, whatever an LLM should know first.\n\n## Docs\n\n- [Getting started](https://example.com/docs/start): the five-minute setup\n- [API reference](https://example.com/docs/api): every endpoint, typed\n\n## Guides\n\n- [Deploying to production](https://example.com/guides/deploy)\n- [Troubleshooting TLS](https://example.com/guides/tls)\n\n## Optional\n\n- [Changelog](https://example.com/changelog)\n```\n\nThe pieces that matter:\n\n`>`\n\n) immediately after, holding a short summary. Agents are told to treat this as the elevator pitch.`: a short description`\n\nof what's behind it.`## Optional`\n\nsectionThere's also a companion file, ** llms-full.txt**, for sites that want to inline the\n\n`llms.txt`\n\nlinks; `llms-full.txt`\n\nincludes. Most sites only need the former.The failure mode isn't a syntax error — it's an `llms.txt`\n\nthat's technically valid and practically useless because it lists your blog's forty most recent posts, or duplicates your nav. Curate it the way you'd brief a smart stranger:\n\n`[Pricing](…): four tiers from free to $699/mo, no per-seat billing`\n\nis a citable fact; a bare link isn't.This is the question that matters and the one most \"add an llms.txt today!\" posts skip, so here it is straight: **adoption on the publishing side is real and growing; consumption by the major answer engines is not something anyone can currently guarantee.** OpenAI, Anthropic, Google, and Perplexity have not committed to reading `llms.txt`\n\nat retrieval time the way they commit to honoring `robots.txt`\n\n. Some developer-facing tools and agent frameworks do look for it. The big consumer answer engines mostly still crawl and parse your rendered pages directly.\n\nSo why add one? Three reasons that hold regardless of whether ChatGPT reads it this quarter:\n\nWhat you should *not* do is treat it as a substitute for the things engines demonstrably *do* use today — being crawlable (robots.txt + no WAF rule silently 403-ing the answer engines) and being parseable (JSON-LD structured data). `llms.txt`\n\nis the third layer, not the first. If an answer engine can't fetch your pages at all, a perfect `llms.txt`\n\nchanges nothing.\n\nGetting cited by an AI answer engine is three questions in order, and `llms.txt`\n\nis only the last one:\n\n`robots.txt`\n\n`llms.txt`\n\n— the curated table of contents.Skip step 1 and the other two are moot. Nail all three and you've done everything a site can currently do to be legible to an AI agent.\n\n\"Present\" and \"valid\" are different. A file that returns `200`\n\nbut is empty, or is a wall of prose with no headings or links, is present-but-useless — and an agent parsing it as Markdown gets nothing structured out of it. Verify it the way a consumer would:\n\n```\n   curl -s -o /dev/null -w \"%{http_code}\\n\" https://example.com/llms.txt\n```\n\nA `404`\n\nmeans you don't have one. A `200`\n\nthat serves your HTML error page (a SPA catch-all route) is worse — it \"exists\" but is garbage.\n\n`llms.txt`\n\nhas at least one heading (`#`\n\n) or one Markdown link (`[text](url)`\n\n). If it has neither, a parser can't extract structure from it.If you'd rather not eyeball it, the free [AI Agent-Readiness checker](https://merlonix.com/tools/agent-readiness/) does exactly this from outside your stack: it fetches your `/llms.txt`\n\n, checks that it's present *and* shaped like real Markdown (a heading or a link — not an empty file or an error page served in its place), and folds that into a 0–100 score alongside the two things that matter more today — whether your `robots.txt`\n\nactually lets the answer-engine crawlers in, and whether your pages carry JSON-LD structured data. It tells you which of the three layers is your weak point, which is the only question worth acting on. No signup, one domain at a time.\n\nThe one-line version: ** llms.txt is a curated, Markdown table of contents you write for AI agents — cheap to add, standards-track, and legible, but it is not access control and it is not yet something the major answer engines promise to read.** Add it because it's a near-free option on a likely future and a canonical statement of your key pages — but add it\n\nMerlonix watches all three layers the way it watches SSL, DNS, and domain expiry: continuously, and from outside your infrastructure, so a redeploy that drops your `llms.txt`\n\n, a new WAF rule that 403s the answer engines, or a template change that strips your JSON-LD doesn't quietly erode your AI-answer visibility weeks before anyone notices the referral traffic fall off. Run the free [agent-readiness scan](https://merlonix.com/tools/agent-readiness/) to see where a domain stands today, [check its live SSL and DNS](https://merlonix.com/tools/domain-health/) while you're there, and browse the rest of the [free tools](https://merlonix.com/tools/). Being findable by an agent is a configuration problem you can mostly fix this afternoon — once you know which of the three layers is the one holding you back.", "url": "https://wpnews.pro/news/llms-txt-explained-what-it-is-whether-ai-answer-engines-read-it-and-how-to-write", "canonical_source": "https://dev.to/merlonix/llmstxt-explained-what-it-is-whether-ai-answer-engines-read-it-and-how-to-write-one-5gof", "published_at": "2026-08-24 20:10:02+00:00", "updated_at": "2026-08-24 20:43:43.141001+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools"], "entities": ["Merlonix", "llmstxt.org"], "alternates": {"html": "https://wpnews.pro/news/llms-txt-explained-what-it-is-whether-ai-answer-engines-read-it-and-how-to-write", "markdown": "https://wpnews.pro/news/llms-txt-explained-what-it-is-whether-ai-answer-engines-read-it-and-how-to-write.md", "text": "https://wpnews.pro/news/llms-txt-explained-what-it-is-whether-ai-answer-engines-read-it-and-how-to-write.txt", "jsonld": "https://wpnews.pro/news/llms-txt-explained-what-it-is-whether-ai-answer-engines-read-it-and-how-to-write.jsonld"}}