{"slug": "llms-txt-what-it-actually-does-and-why-it-rots", "title": "llms.txt: What It Actually Does, and Why It Rots", "summary": "A developer explains that llms.txt, a markdown file that optimizes site content for LLM retrieval, is often implemented poorly and rots over time. They built a free generator, aicrawlable.com, to automate validation and prevent stale descriptions and broken links.", "body_md": "When ChatGPT, Claude or Perplexity answers a question about your product, it is not consulting a decade of PageRank. It fetches a handful of pages and tries to work out what your site is. That is a very different retrieval problem from classic search, and most sites are accidentally hostile to it.\n\nA sitemap optimises for **coverage** — every URL, every paginated archive, every tag page. That is correct for a crawler with a huge budget and a ranking model to sort the noise afterwards.\n\nAn LLM landing on your site has neither. It has a limited context window and one shot. If the first thing it ingests is 400 URLs of `?page=17`\n\nand `/tag/misc`\n\n, your three genuinely useful guides are buried.\n\n[llms.txt](https://llmstxt.org) inverts this. It is a small markdown file at your root that optimises for **priority**:\n\n```\n# Your Product\n\n> One-line description of what this actually does.\n\n## Docs\n- [Quickstart](https://example.com/docs/quickstart): Install and first request in 5 minutes\n- [API Reference](https://example.com/docs/api): Every endpoint with request/response examples\n```\n\nTwo rules make or break it:\n\n`llms-full.txt`\n\ninlines expanded content rather than linking out, so a model can ingest everything in one request. This is genuinely useful for compact docs — and actively harmful for large sites, where you will blow the context window and get truncated mid-document.\n\nRough heuristic: if your docs exceed roughly 50k tokens, ship `llms.txt`\n\nalone and let models fetch selectively.\n\nThis is where most implementations quietly fail. You write the file, ship it, and three months later half the descriptions describe features you renamed and two links 404. Nothing in your build catches it, because nothing validates it.\n\nTwo things worth automating:\n\n`llms.txt`\n\nstill returns 200I ended up building a free generator for exactly this loop, after my own docs got cited with a description I had rewritten months earlier. Paste a URL, it crawls the structure, and produces spec-compliant `llms.txt`\n\nand `llms-full.txt`\n\nyou can edit before exporting — reorder, drop low-value pages, rewrite descriptions. No signup, nothing stored: [aicrawlable.com](https://aicrawlable.com).\n\nNo search engine has committed to llms.txt as a ranking signal, and it may never become one. What it does today is cheap insurance: when a model *does* fetch your site, it reads a curated map instead of guessing. That asymmetry — near-zero cost, meaningful upside — is the whole argument.", "url": "https://wpnews.pro/news/llms-txt-what-it-actually-does-and-why-it-rots", "canonical_source": "https://dev.to/danielgreid/llmstxt-what-it-actually-does-and-why-it-rots-hh", "published_at": "2026-07-24 18:46:10+00:00", "updated_at": "2026-07-24 19:02:23.481948+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": ["ChatGPT", "Claude", "Perplexity", "aicrawlable.com"], "alternates": {"html": "https://wpnews.pro/news/llms-txt-what-it-actually-does-and-why-it-rots", "markdown": "https://wpnews.pro/news/llms-txt-what-it-actually-does-and-why-it-rots.md", "text": "https://wpnews.pro/news/llms-txt-what-it-actually-does-and-why-it-rots.txt", "jsonld": "https://wpnews.pro/news/llms-txt-what-it-actually-does-and-why-it-rots.jsonld"}}