Originally published on the Merlonix blog.
/llms.txt
is the file everyone adding "AI SEO" to their site keeps hearing about and nobody quite explains. It gets lumped in with robots.txt
because it lives in the same place and has a similar name, so people assume it's another access-control file — a way to let the good AI crawlers in or keep them out. It is not. robots.txt
says who may fetch what. llms.txt
says here is what matters, and where it is. One is a gate; the other is a table of contents. Adding an llms.txt
grants no access and blocks nothing — a crawler that a WAF or robots.txt rule is keeping out will never see it, and a crawler that's already allowed in doesn't need your permission to read it. What it does is make a site legible to an agent that has to otherwise reverse-engineer your navigation to find the three pages that actually answer the question.
Here's the whole thing: what the file is, the exact format, what to put in it, the honest answer to "does anything actually read it yet," and how to confirm yours is valid rather than just present.
llms.txt
is a proposed convention — published at llmstxt.org in late 2024 — for a single Markdown file at your domain root (https://example.com/llms.txt
) that curates your most important content for a large language model. The premise is simple: an LLM working with a limited context window can't crawl and hold your entire site, and rendered marketing HTML is noisy — nav, cookie banners, footers, scripts. A short, hand-curated Markdown file that names your key pages and links straight to them is far cheaper for an agent to consume than making it guess which of your 400 URLs are the canonical docs.
It is not a standard in the RFC sense — it's a community convention with growing adoption, mostly among documentation sites and developer tools. It is also not:
robots.txt
llms.txt
is curation and grants nothing.sitemap.xml
is an exhaustive, machine-generated list of llms.txt
is the opposite: short, hand-picked, human-written, and llms.txt
says "here are the six things that matter."this is an Article
, this is a Product
). llms.txt
operates at the site level, pointing across pages.Think of it as the README you'd hand a new hire who has ten minutes: not the whole codebase, just "start here, then here, and here's where pricing lives."
The spec is deliberately minimal, and it's just Markdown, so it renders fine for a human too. The shape:
> An optional one-line blockquote summary of what this
> site or project is, so an agent gets the gist immediately.
Some optional free-form Markdown here — a paragraph of
context, key facts, whatever an LLM should know first.
## Docs
- [Getting started](https://example.com/docs/start): the five-minute setup
- [API reference](https://example.com/docs/api): every endpoint, typed
## Guides
- [Deploying to production](https://example.com/guides/deploy)
- [Troubleshooting TLS](https://example.com/guides/tls)
## Optional
- [Changelog](https://example.com/changelog)
The pieces that matter:
>
) immediately after, holding a short summary. Agents are told to treat this as the elevator pitch.: a short description
of what's behind it.## Optional
sectionThere's also a companion file, ** llms-full.txt**, for sites that want to inline the
llms.txt
links; llms-full.txt
includes. Most sites only need the former.The failure mode isn't a syntax error — it's an llms.txt
that's technically valid and practically useless because it lists your blog's forty most recent posts, or duplicates your nav. Curate it the way you'd brief a smart stranger:
[Pricing](…): four tiers from free to $699/mo, no per-seat billing
is a citable fact; a bare link isn't.This is the question that matters and the one most "add an llms.txt today!" posts skip, so here it is straight: adoption on the publishing side is real and growing; consumption by the major answer engines is not something anyone can currently guarantee. OpenAI, Anthropic, Google, and Perplexity have not committed to reading llms.txt
at retrieval time the way they commit to honoring robots.txt
. Some developer-facing tools and agent frameworks do look for it. The big consumer answer engines mostly still crawl and parse your rendered pages directly.
So why add one? Three reasons that hold regardless of whether ChatGPT reads it this quarter:
What you should not do is treat it as a substitute for the things engines demonstrably do use today — being crawlable (robots.txt + no WAF rule silently 403-ing the answer engines) and being parseable (JSON-LD structured data). llms.txt
is the third layer, not the first. If an answer engine can't fetch your pages at all, a perfect llms.txt
changes nothing.
Getting cited by an AI answer engine is three questions in order, and llms.txt
is only the last one:
robots.txt
llms.txt
— the curated table of contents.Skip step 1 and the other two are moot. Nail all three and you've done everything a site can currently do to be legible to an AI agent.
"Present" and "valid" are different. A file that returns 200
but is empty, or is a wall of prose with no headings or links, is present-but-useless — and an agent parsing it as Markdown gets nothing structured out of it. Verify it the way a consumer would:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/llms.txt
A 404
means you don't have one. A 200
that serves your HTML error page (a SPA catch-all route) is worse — it "exists" but is garbage.
llms.txt
has at least one heading (#
) or one Markdown link ([text](url)
). If it has neither, a parser can't extract structure from it.If you'd rather not eyeball it, the free AI Agent-Readiness checker does exactly this from outside your stack: it fetches your /llms.txt
, checks that it's present and shaped like real Markdown (a heading or a link — not an empty file or an error page served in its place), and folds that into a 0–100 score alongside the two things that matter more today — whether your robots.txt
actually lets the answer-engine crawlers in, and whether your pages carry JSON-LD structured data. It tells you which of the three layers is your weak point, which is the only question worth acting on. No signup, one domain at a time.
The one-line version: ** llms.txt is a curated, Markdown table of contents you write for AI agents — cheap to add, standards-track, and legible, but it is not access control and it is not yet something the major answer engines promise to read.** Add it because it's a near-free option on a likely future and a canonical statement of your key pages — but add it
Merlonix watches all three layers the way it watches SSL, DNS, and domain expiry: continuously, and from outside your infrastructure, so a redeploy that drops your llms.txt
, a new WAF rule that 403s the answer engines, or a template change that strips your JSON-LD doesn't quietly erode your AI-answer visibility weeks before anyone notices the referral traffic fall off. Run the free agent-readiness scan to see where a domain stands today, check its live SSL and DNS while you're there, and browse the rest of the free tools. Being findable by an agent is a configuration problem you can mostly fix this afternoon — once you know which of the three layers is the one holding you back.