{"slug": "llms-txt-llms-full-txt-teaching-llms-what-your-site-offers", "title": "llms.txt & llms-full.txt: Teaching LLMs What Your Site Offers", "summary": "A new web standard called llms.txt and its companion llms-full.txt provide a machine-readable markdown file at a site's root, giving AI agents a curated map of key pages to reduce token costs and improve accuracy. The spec, hosted at llmstxt.org, aims to replace inefficient HTML crawling with a single, high-signal read for agent discovery and task completion.", "body_md": "*Part of the Agent Readiness course — the web standards that decide whether an AI agent can actually read, understand, and act on your site. Measure any page against these with the Core Agent Vitals analyzer.*\n\n## What it is\n\n`llms.txt`\n\nis a single markdown file you publish at your site root — `https://your-site.com/llms.txt`\n\n. It gives LLMs and AI agents a **curated, machine-readable map** of your site: your name, a one-line description, and the handful of links that actually matter (docs, pricing, API, key articles), each with a short note on what it contains.\n\nA companion file, `llms-full.txt`\n\n, goes further: it inlines the **actual content** of those key pages as clean markdown, so an agent can answer questions about your site from one fetch — no crawling, no HTML parsing, no JavaScript execution.\n\nThink of `llms.txt`\n\nas the table of contents and `llms-full.txt`\n\nas the printed book. The spec lives at [llmstxt.org](https://llmstxt.org).\n\n## Why agents need it\n\nAn AI agent works in a loop: **observe → reason → act**. The quality of everything downstream depends on that first observation. Without a machine-readable index, the agent's only option is to fetch your homepage HTML, strip the navigation and boilerplate, guess which links are relevant, fetch those, and repeat — spending tokens and time on markup it will throw away, and frequently landing on the wrong page.\n\n`llms.txt`\n\ncollapses that whole discovery phase into one cheap, high-signal read:\n\n**Lower token cost.** A 200-line markdown index costs a fraction of rendering and tokenizing full HTML pages. On the analyzer this shows up directly in**Token Cost (TC)**.** Fewer wrong answers.**You decide what the agent sees first, so it stops inferring your structure from a cluttered DOM.** Deterministic entry points.**Agents that support the convention read`/llms.txt`\n\nbefore crawling — you get to curate the first impression.\n\n## How to implement\n\n**1. Create /llms.txt** — an H1 with your name, a blockquote one-liner, then link sections:\n\n```\n> Developer documentation and API reference for the Acme platform.\n\n## Docs\n- [Quickstart](https://acme.com/docs/quickstart): Install and make your first call in 5 minutes.\n- [Authentication](https://acme.com/docs/auth): API keys, OAuth, and scopes.\n\n## API\n- [REST reference](https://acme.com/api): All endpoints, params, and response shapes.\n\n## Optional\n- [Changelog](https://acme.com/changelog): Release notes.\n```\n\n**2. (Recommended) Create /llms-full.txt** — the same structure, but paste the clean markdown content of each key page inline. Generate it from your existing markdown/MDX at build time so it never drifts.\n\n**3. Serve both as text/plain at the root.** On a static host, drop the files in\n\n`public/`\n\n. On a framework, add two routes that return the files with `Content-Type: text/plain; charset=utf-8`\n\n.## Validate\n\n```\ncurl -s https://your-site.com/llms.txt | head\n```\n\nYou should get markdown, HTTP 200, and a `text/plain`\n\ncontent type. Then run your site through the [Core Agent Vitals analyzer](https://agentvitals.dev/analyze) — the **Agent Discoverability** panel checks for `llms.txt`\n\nat both `/llms.txt`\n\nand `/.well-known/llms.txt`\n\n, and flags it if the H1/blockquote structure is missing.\n\n## Common mistakes\n\n**Publishing it as HTML.** If your framework wraps it in a layout, agents get a web page, not markdown. Serve raw`text/plain`\n\n.**Listing every URL.**`llms.txt`\n\nis a*curated*index, not a sitemap. If it lists 400 links it's noise — link the 5–15 pages that matter and let the sitemap handle the long tail.**No H1 or blockquote.** Parsers key off the`# Title`\n\nand`> one-liner`\n\n. Skip them and tools treat the file as malformed — it fails silently.**Letting** If you hand-maintain the inlined content, it drifts from the real pages within weeks. Generate it from source at build time or don't ship it.`llms-full.txt`\n\nrot.**Blocking the crawlers that would read it.** An`llms.txt`\n\nbehind a`robots.txt`\n\ndisallow, a login wall, or a bot-challenge is invisible. Keep the file and the pages it points to publicly reachable.\n\n*Next in the course: Sitemaps for Agent Discovery — the table of contents for everything llms.txt doesn't curate.*", "url": "https://wpnews.pro/news/llms-txt-llms-full-txt-teaching-llms-what-your-site-offers", "canonical_source": "https://blog.r-lopes.com/posts/agent-readiness-llms-txt", "published_at": "2026-07-02 14:00:00+00:00", "updated_at": "2026-07-03 21:15:24.522273+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-tools", "ai-infrastructure"], "entities": ["llms.txt", "llms-full.txt", "llmstxt.org", "Core Agent Vitals"], "alternates": {"html": "https://wpnews.pro/news/llms-txt-llms-full-txt-teaching-llms-what-your-site-offers", "markdown": "https://wpnews.pro/news/llms-txt-llms-full-txt-teaching-llms-what-your-site-offers.md", "text": "https://wpnews.pro/news/llms-txt-llms-full-txt-teaching-llms-what-your-site-offers.txt", "jsonld": "https://wpnews.pro/news/llms-txt-llms-full-txt-teaching-llms-what-your-site-offers.jsonld"}}