When ChatGPT, Claude or Perplexity answers a question about your product, it is not consulting a decade of PageRank. It fetches a handful of pages and tries to work out what your site is. That is a very different retrieval problem from classic search, and most sites are accidentally hostile to it.
A sitemap optimises for coverage — every URL, every paginated archive, every tag page. That is correct for a crawler with a huge budget and a ranking model to sort the noise afterwards.
An LLM landing on your site has neither. It has a limited context window and one shot. If the first thing it ingests is 400 URLs of ?page=17
and /tag/misc
, your three genuinely useful guides are buried.
llms.txt inverts this. It is a small markdown file at your root that optimises for priority:
> One-line description of what this actually does.
## Docs
- [Quickstart](https://example.com/docs/quickstart): Install and first request in 5 minutes
- [API Reference](https://example.com/docs/api): Every endpoint with request/response examples
Two rules make or break it:
llms-full.txt
inlines expanded content rather than linking out, so a model can ingest everything in one request. This is genuinely useful for compact docs — and actively harmful for large sites, where you will blow the context window and get truncated mid-document.
Rough heuristic: if your docs exceed roughly 50k tokens, ship llms.txt
alone and let models fetch selectively.
This is where most implementations quietly fail. You write the file, ship it, and three months later half the descriptions describe features you renamed and two links 404. Nothing in your build catches it, because nothing validates it.
Two things worth automating:
llms.txt
still returns 200I ended up building a free generator for exactly this loop, after my own docs got cited with a description I had rewritten months earlier. Paste a URL, it crawls the structure, and produces spec-compliant llms.txt
and llms-full.txt
you can edit before exporting — reorder, drop low-value pages, rewrite descriptions. No signup, nothing stored: aicrawlable.com.
No search engine has committed to llms.txt as a ranking signal, and it may never become one. What it does today is cheap insurance: when a model does fetch your site, it reads a curated map instead of guessing. That asymmetry — near-zero cost, meaningful upside — is the whole argument.