{"slug": "llms-txt-what-it-is-and-how-to-add-one", "title": "llms.txt: what it is and how to add one", "summary": "Lyra published a guide explaining the llms.txt file, a plain Markdown file hosted at /llms.txt that provides AI crawlers with a curated map of a site's most important pages. Proposed by Jeremy Howard in 2024, the file includes the site name, a summary, and links with descriptions, helping language models understand site context without parsing raw HTML. The guide covers the file's structure, its differences from robots.txt and sitemap.xml, and practical advice for implementation.", "body_md": "*Originally published on the Lyra blog.*\n\nAn llms.txt file is a plain Markdown file you host at `/llms.txt`\n\nthat hands AI crawlers a clean, curated map of your site's most important pages. It is not styled for people. It is a shortlist a language model can read in one pass: your site name, a one-paragraph summary, and a few sections of links with short descriptions. The idea, proposed by Jeremy Howard in 2024, is simple. Models work better with context than with raw HTML, so give them the context directly.\n\nThis guide covers what goes in an llms.txt file, whether it actually moves the needle, how it differs from robots.txt and a sitemap, and how to write and host one in a few minutes.\n\nllms.txt is a Markdown file at the root of your domain that points AI models at the pages that matter, with a sentence of context on each. A browser will render it as text. A model reading it gets a quick, structured sense of what your site is and where the good parts live, without crawling and parsing every page first.\n\nThe format is loose but conventional. A well-formed file has four parts:\n\nBecause it is just Markdown, it reads the same way structured content reads to a model: predictable headings, plain prose, no markup noise. That predictability is the point. The same instinct drives good [answer engine optimization](https://www.trylyra.ai/blog/answer-engine-optimization/): answer the question plainly, structure it cleanly, and machine readers pick it up. An llms.txt file applies that thinking at the level of your whole site instead of a single page.\n\nIt pairs with the technical hygiene that already helps you rank. If you have worked through [SEO for SaaS](https://www.trylyra.ai/blog/seo-for-saas/), an llms.txt file is a small, fast addition to that same checklist, not a replacement for it.\n\nHonest answer: it is an emerging convention, not a ranking guarantee. No major AI provider has publicly confirmed that an llms.txt file changes how they rank, retrieve, or cite your pages. Adoption among the big crawlers is still uneven, and some teams treat it with healthy skepticism for exactly that reason.\n\nSo why add one? A few reasons that hold up even without a confirmed ranking benefit:\n\nTreat it the way you would treat clean structured data or a tidy sitemap: good practice with a low ceiling on downside, not a magic lever. The win is not the file alone. It is the habit of writing for both people and the models that increasingly sit between people and your site. The same logic is why we [fact-check every claim and link](https://www.trylyra.ai/blog/ai-content-fact-checking/) in a post before it ships: if a model is going to cite you, the page it cites had better be accurate and easy to read.\n\nIf you want to see a live one, Lyra serves her own at [trylyra.ai/llms.txt](https://www.trylyra.ai/llms.txt). It is a small, real example of the format described here.\n\nThey live in the same place and they all talk to crawlers, but they do different jobs. Mixing them up is the most common confusion, so here is the split.\n\n| File | What it does | Who it is for |\n|---|---|---|\n`robots.txt` |\nSets rules: which paths crawlers may or may not access. | Crawlers deciding what to fetch. |\n`sitemap.xml` |\nLists every URL so crawlers can discover them all. | Search engines indexing the full site. |\n`llms.txt` |\nCurates your best pages with plain-language descriptions. | Language models that want context fast. |\n\nA sitemap is exhaustive. It wants to list everything. An llms.txt file is the opposite: it is opinionated and short, a highlight reel of the pages you would actually want an AI to read and quote. robots.txt is about permission, not content: it decides which crawlers may read you, and getting the [AI crawler rules right](https://www.trylyra.ai/blog/robots-txt-for-ai-bots/) is its own job, because blocking the wrong bot quietly deletes you from AI answers. None of the three replaces the others. You can and usually should have all three.\n\nOne nuance worth keeping straight. Some teams also publish full-text versions of pages at paths like `page.md`\n\nor an `llms-full.txt`\n\nthat inlines whole documents. That is an extension, not a requirement. The core llms.txt file is just the map.\n\nStart with the pages you would hand a new hire on day one. Your homepage, your core product or pricing pages, your docs, and a handful of your strongest blog posts. Skip thin pages, duplicates, and anything expired. Quality over coverage. A model gets more from 15 well-described links than from 150 bare URLs. Your docs pages deserve the same scrutiny as the rest of the list: they carry the highest concentration of AI agent traffic on most sites, so getting them structured for [docs SEO](https://www.trylyra.ai/blog/docs-seo-getting-your-api-documentation-cited-by-ai/) matters at least as much as which links make the file.\n\nThen write the file. Here is a minimal example you can adapt:\n\n```\n# Acme Analytics\n\n> Acme Analytics is a privacy-first product analytics tool for small SaaS\n> teams. We help founders see which features drive retention without\n> shipping user data to third parties.\n\nSelf-serve, no credit card to start. Docs are public and versioned.\n\n## Product\n\n- [Overview](https://acme.com/): What Acme does and who it is for.\n- [Pricing](https://acme.com/pricing/): Plans, limits, and the free tier.\n- [Integrations](https://acme.com/integrations/): Supported sources and SDKs.\n\n## Docs\n\n- [Quickstart](https://acme.com/docs/quickstart/): Install and send your first event.\n- [API reference](https://acme.com/docs/api/): Endpoints, auth, and rate limits.\n\n## Blog\n\n- [Retention metrics that matter](https://acme.com/blog/retention/): The four numbers we track.\n- [Self-hosting Acme](https://acme.com/blog/self-host/): Run it on your own infra.\n```\n\nA few rules that keep it useful:\n\nThe file has to resolve at your domain root, at `https://yourdomain.com/llms.txt`\n\n, and return plain text or Markdown. How you get it there depends on your stack.\n\n`llms.txt`\n\nin your `public/`\n\nor root output directory. Most static hosts serve it as-is.`text/plain`\n\nor `text/markdown`\n\ncontent type.Once it is live, open the URL in a browser and confirm you see your raw Markdown, not a 404 and not an HTML page wrapping it. That is the whole job. There is no registry to submit to and no verification step.\n\nAn llms.txt file is one small piece. It makes your site easy to map. It does not write the pages that make the map worth reading. Those still come from clear, accurate, well-structured content that answers real questions, which is what gets cited whether or not a crawler reads your llms.txt first. The file is a pointer; the pages are the payload.\n\nThis is where the work scales or stalls. Maintaining the file is trivial. Producing a steady stream of pages worth pointing to is the hard part, and it is exactly the gap Lyra fills. Lyra is an autonomous writer who finds topics worth covering, drafts them in your blog's existing voice, fact-checks the claims and links, scores the draft, and opens a GitHub pull request for you to review. Nothing publishes on its own. You bring your own Anthropic key, and Lyra is in early access while we build in the open, so [talk to the founder](https://www.trylyra.ai/contact/) to see whether she's a fit. If you want the deeper version of how an [autonomous AI blog writer](https://dev.to/ai-blog-writer/) works end to end, the pillar page walks through it.\n\nAdd the llms.txt file this afternoon. It is five minutes of work with no real downside. Then spend your energy on the content it points to, because that is what actually earns the citation.\n\n*I'm building Lyra, an autonomous blog writer that writes in your blog's voice, fact-checks every claim, and opens a pull request you review. This post comes from her blog, where we publish what we learn running the pipeline. Happy to answer questions in the comments.*", "url": "https://wpnews.pro/news/llms-txt-what-it-is-and-how-to-add-one", "canonical_source": "https://dev.to/mitrasish/llmstxt-what-it-is-and-how-to-add-one-4g3k", "published_at": "2026-07-07 20:30:00+00:00", "updated_at": "2026-07-07 20:58:25.454107+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-products"], "entities": ["Lyra", "Jeremy Howard", "llms.txt", "trylyra.ai"], "alternates": {"html": "https://wpnews.pro/news/llms-txt-what-it-is-and-how-to-add-one", "markdown": "https://wpnews.pro/news/llms-txt-what-it-is-and-how-to-add-one.md", "text": "https://wpnews.pro/news/llms-txt-what-it-is-and-how-to-add-one.txt", "jsonld": "https://wpnews.pro/news/llms-txt-what-it-is-and-how-to-add-one.jsonld"}}