llms.txt: What It Actually Does, and Why It Rots

A developer explains that llms.txt, a markdown file that optimizes site content for LLM retrieval, is often implemented poorly and rots over time. They built a free generator, aicrawlable.com, to automate validation and prevent stale descriptions and broken links.

When ChatGPT, Claude or Perplexity answers a question about your product, it is not consulting a decade of PageRank. It fetches a handful of pages and tries to work out what your site is. That is a very different retrieval problem from classic search, and most sites are accidentally hostile to it. A sitemap optimises for coverage — every URL, every paginated archive, every tag page. That is correct for a crawler with a huge budget and a ranking model to sort the noise afterwards. An LLM landing on your site has neither. It has a limited context window and one shot. If the first thing it ingests is 400 URLs of ?page=17 and /tag/misc , your three genuinely useful guides are buried. llms.txt https://llmstxt.org inverts this. It is a small markdown file at your root that optimises for priority : Your Product One-line description of what this actually does. Docs - Quickstart https://example.com/docs/quickstart : Install and first request in 5 minutes - API Reference https://example.com/docs/api : Every endpoint with request/response examples Two rules make or break it: llms-full.txt inlines expanded content rather than linking out, so a model can ingest everything in one request. This is genuinely useful for compact docs — and actively harmful for large sites, where you will blow the context window and get truncated mid-document. Rough heuristic: if your docs exceed roughly 50k tokens, ship llms.txt alone and let models fetch selectively. This is where most implementations quietly fail. You write the file, ship it, and three months later half the descriptions describe features you renamed and two links 404. Nothing in your build catches it, because nothing validates it. Two things worth automating: llms.txt still returns 200I ended up building a free generator for exactly this loop, after my own docs got cited with a description I had rewritten months earlier. Paste a URL, it crawls the structure, and produces spec-compliant llms.txt and llms-full.txt you can edit before exporting — reorder, drop low-value pages, rewrite descriptions. No signup, nothing stored: aicrawlable.com https://aicrawlable.com . No search engine has committed to llms.txt as a ranking signal, and it may never become one. What it does today is cheap insurance: when a model does fetch your site, it reads a curated map instead of guessing. That asymmetry — near-zero cost, meaningful upside — is the whole argument.