{"slug": "llms-txt-is-not-a-docs-url-dump", "title": "Llms.txt is not a docs url dump", "summary": "Most llms.txt files fail because they include too many links and lack editorial judgment, treating the file as a complete documentation dump rather than a curated task map for AI agents. The file's purpose is to narrow an agent's search space, so teams should include only pages that directly help with tasks like first-time setup, authentication, error fixing, or core API usage. Marketing pages, changelog noise, outdated content, and duplicate versioned docs should be excluded to keep the file focused and useful.", "body_md": "**Most bad llms.txt files fail for a boring reason: they are trying too hard to be complete.** They include too many links, too much adjacent content, and not enough judgment.\n\nThat usually comes from a reasonable instinct. Nobody wants to leave out something important. But `llms.txt`\n\nis one of those cases where completeness is not the goal.\n\nThe job of the file is to narrow the search space for an agent.\n\nSo the real question is not \"what can we include?\" It is \"what should an agent read first if it is trying to actually use this product?\"\n\nIf you have not read the basics yet, start with [What Is llms.txt? A Practical Guide for SaaS Docs Teams](/blog/what-is-llms-txt). If you are deciding how to structure the file mechanically, read\n\n[How to Add](/blog/how-to-add-llms-txt-to-your-documentation-site).\n\n`llms.txt`\n\nto Your Documentation SiteThis post is about the harder editorial question: what belongs in the file, and what should stay out?\n\n## Start From the Job the Docs Need to Do\n\nFor most SaaS products, an agent using your docs is trying to do one of a few things:\n\n- complete first-time setup\n- authenticate correctly\n- call the right endpoint or use the right SDK\n- fix an error\n- configure an integration\n- understand a concept that blocks implementation\n\nThat should shape the file.\n\n`llms.txt`\n\nis not a brand surface. It is not a docs directory. It is not a page inventory. It is a curated task map.\n\nIf a page does not materially help one of those tasks, it probably does not belong in the first version.\n\n## What to Include\n\nFor most SaaS docs teams, the default keep-list is pretty stable.\n\n### 1. The real quickstart\n\nNot the page you call the quickstart. The page that actually gets someone to a successful first integration.\n\nIf your current quickstart is mostly marketing framing or architectural background, do not pretend it is the best first step. Use the page that gets the work done.\n\n### 2. Authentication\n\nThis is almost always worth including.\n\nIf your product supports more than one auth model, make the recommended path obvious.\n\n### 3. Core API or product entry point\n\nFor API docs, that might be the main API overview or a reference landing page.\n\nFor product docs, it might be the setup overview or the main workflow page.\n\n### 4. Errors, rate limits, retries, or troubleshooting\n\nThese are disproportionately useful for agents because they show up mid-task, not just at the start.\n\n### 5. Webhooks, SDKs, or integrations that represent real adoption paths\n\nInclude the things people actually use, not every possible integration surface.\n\n### 6. One or two concept pages if the product has real architecture\n\nIf your docs require understanding environments, tenancy, sync models, or permissions before implementation makes sense, include those pages explicitly.\n\n## What to Skip\n\nThis is where most files get noisy.\n\n### 1. Marketing pages\n\nDo not include:\n\n- product overview pages written for buyers\n- pricing\n- generic landing pages\n- announcement posts\n\nEven if those pages are technically part of the same site, they do not usually help an agent solve a docs task.\n\n### 2. Changelog noise\n\nA changelog can be useful, but most products should not treat it as core `llms.txt`\n\nmaterial.\n\nIf one release introduced a breaking auth change or a new required integration path, maybe that deserves a mention elsewhere. But a long release note archive should not dominate the file.\n\n### 3. Thin or outdated pages\n\nIf you would not send a new engineer there, do not send an agent there either.\n\n`llms.txt`\n\nis not the place to preserve internal guilt about docs debt.\n\n### 4. Duplicate versioned docs\n\nIf you have multiple versions of the same guide, choose the canonical one unless older versions are still active and materially different.\n\n### 5. Internal tools, dashboards, or auth-gated pages\n\nThis sounds obvious, but it is worth saying. The file should reflect the useful public docs surface.\n\n### 6. Giant link lists with no structure\n\nEven if every page is individually valid, a wall of links is still a weak file.\n\n## The Useful Test: Would You Hand This to a New Engineer?\n\nThis is probably the best editorial test I know for `llms.txt`\n\n.\n\nOpen the file and ask:\n\n**If a competent engineer joined the team today, would this be a good starting map?**\n\nIf the answer is yes, you are probably close.\n\nIf the answer is no, the problem is usually one of three things:\n\n- too many links\n- the wrong links\n- groupings that reflect internal org charts instead of user tasks\n\nThat test is simple, but it catches most of the mistakes.\n\n## What Different SaaS Docs Surfaces Usually Need\n\nNot every docs site should include the same categories.\n\n### API docs\n\nBias toward:\n\n- quickstart\n- authentication\n- API overview\n- errors\n- rate limits\n- webhooks\n- primary SDKs\n\n### Help center\n\nBias toward:\n\n- getting started\n- workspace or account setup\n- permissions\n- billing\n- high-volume troubleshooting\n- key integrations\n\n### Developer platform docs\n\nBias toward:\n\n- quickstart\n- local development\n- CLI\n- deployment\n- architecture or data model\n- auth\n- SDKs\n\nThe point is not to force one universal structure. The point is to choose the pages that fit the actual job of the docs surface.\n\n## Pages That Feel Important but Usually Do Not Help\n\nSome pages get included because they feel official.\n\nExamples:\n\n- glossary pages\n- generic \"overview\" pages that say very little\n- category landing pages with no real content\n- press-announcement-style release posts\n- \"all integrations\" directories\n\nThese are not always bad pages. They are just usually not high-leverage pages for agent guidance.\n\nAn agent benefits more from one strong webhook guide than from a top-level \"Integrations\" page that links to twenty products.\n\n## A Good File Usually Has a Point of View\n\nThis is the subtle part.\n\nThe best `llms.txt`\n\nfiles are not neutral. They express a recommendation:\n\n- start here\n- this is the canonical auth path\n- these are the common failure modes\n- this SDK is the usual default\n- this concept matters before you implement anything else\n\nThat point of view is what makes the file helpful.\n\nWithout it, you still have documentation. You just do not have guidance.\n\nThat is also why auto-generated `llms.txt`\n\nfiles tend to feel mediocre. Automation is great at collecting links. It is bad at deciding which few links matter most.\n\n## A Practical Keep-or-Skip Checklist\n\nWhen you are deciding whether a page belongs in `llms.txt`\n\n, ask:\n\n- Does this page help someone complete a real implementation or support task?\n- Is this page canonical, current, and worth trusting?\n- Would I want an agent to read this before browsing the rest of the docs?\n- Does it reduce ambiguity, or add more of it?\n\nIf the answer is mostly no, leave it out.\n\nThat is not a failure. It is the point of curation.\n\n## The Short Version\n\nFor SaaS docs, a good `llms.txt`\n\nusually includes:\n\n- quickstart\n- authentication\n- core task pages\n- operational docs like errors and rate limits\n- key SDK or integration paths\n- concept pages that unblock implementation\n\nAnd it usually skips:\n\n- marketing pages\n- changelog noise\n- stale docs\n- duplicate versions\n- auth-gated surfaces\n- giant undifferentiated link dumps\n\nThe format is easy.\n\nThe real value comes from deciding what deserves to be there at all.\n\nThat is why `llms.txt`\n\nis less like a machine-generated inventory and more like a small editorial artifact for your docs system.", "url": "https://wpnews.pro/news/llms-txt-is-not-a-docs-url-dump", "canonical_source": "https://docsalot.dev/blog/llms-txt-for-saas-docs-what-to-include-and-what-to-skip", "published_at": "2026-06-03 23:38:15+00:00", "updated_at": "2026-06-03 23:46:29.958640+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "ai-products", "natural-language-processing"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/llms-txt-is-not-a-docs-url-dump", "markdown": "https://wpnews.pro/news/llms-txt-is-not-a-docs-url-dump.md", "text": "https://wpnews.pro/news/llms-txt-is-not-a-docs-url-dump.txt", "jsonld": "https://wpnews.pro/news/llms-txt-is-not-a-docs-url-dump.jsonld"}}