llms.txt & llms-full.txt: Teaching LLMs What Your Site Offers

A new web standard called llms.txt and its companion llms-full.txt provide a machine-readable markdown file at a site's root, giving AI agents a curated map of key pages to reduce token costs and improve accuracy. The spec, hosted at llmstxt.org, aims to replace inefficient HTML crawling with a single, high-signal read for agent discovery and task completion.

Part of the Agent Readiness course — the web standards that decide whether an AI agent can actually read, understand, and act on your site. Measure any page against these with the Core Agent Vitals analyzer. What it is llms.txt is a single markdown file you publish at your site root — https://your-site.com/llms.txt . It gives LLMs and AI agents a curated, machine-readable map of your site: your name, a one-line description, and the handful of links that actually matter docs, pricing, API, key articles , each with a short note on what it contains. A companion file, llms-full.txt , goes further: it inlines the actual content of those key pages as clean markdown, so an agent can answer questions about your site from one fetch — no crawling, no HTML parsing, no JavaScript execution. Think of llms.txt as the table of contents and llms-full.txt as the printed book. The spec lives at llmstxt.org https://llmstxt.org . Why agents need it An AI agent works in a loop: observe → reason → act . The quality of everything downstream depends on that first observation. Without a machine-readable index, the agent's only option is to fetch your homepage HTML, strip the navigation and boilerplate, guess which links are relevant, fetch those, and repeat — spending tokens and time on markup it will throw away, and frequently landing on the wrong page. llms.txt collapses that whole discovery phase into one cheap, high-signal read: Lower token cost. A 200-line markdown index costs a fraction of rendering and tokenizing full HTML pages. On the analyzer this shows up directly in Token Cost TC . Fewer wrong answers. You decide what the agent sees first, so it stops inferring your structure from a cluttered DOM. Deterministic entry points. Agents that support the convention read /llms.txt before crawling — you get to curate the first impression. How to implement 1. Create /llms.txt — an H1 with your name, a blockquote one-liner, then link sections: Developer documentation and API reference for the Acme platform. Docs - Quickstart https://acme.com/docs/quickstart : Install and make your first call in 5 minutes. - Authentication https://acme.com/docs/auth : API keys, OAuth, and scopes. API - REST reference https://acme.com/api : All endpoints, params, and response shapes. Optional - Changelog https://acme.com/changelog : Release notes. 2. Recommended Create /llms-full.txt — the same structure, but paste the clean markdown content of each key page inline. Generate it from your existing markdown/MDX at build time so it never drifts. 3. Serve both as text/plain at the root. On a static host, drop the files in public/ . On a framework, add two routes that return the files with Content-Type: text/plain; charset=utf-8 . Validate curl -s https://your-site.com/llms.txt | head You should get markdown, HTTP 200, and a text/plain content type. Then run your site through the Core Agent Vitals analyzer https://agentvitals.dev/analyze — the Agent Discoverability panel checks for llms.txt at both /llms.txt and /.well-known/llms.txt , and flags it if the H1/blockquote structure is missing. Common mistakes Publishing it as HTML. If your framework wraps it in a layout, agents get a web page, not markdown. Serve raw text/plain . Listing every URL. llms.txt is a curated index, not a sitemap. If it lists 400 links it's noise — link the 5–15 pages that matter and let the sitemap handle the long tail. No H1 or blockquote. Parsers key off the Title and one-liner . Skip them and tools treat the file as malformed — it fails silently. Letting If you hand-maintain the inlined content, it drifts from the real pages within weeks. Generate it from source at build time or don't ship it. llms-full.txt rot. Blocking the crawlers that would read it. An llms.txt behind a robots.txt disallow, a login wall, or a bot-challenge is invisible. Keep the file and the pages it points to publicly reachable. Next in the course: Sitemaps for Agent Discovery — the table of contents for everything llms.txt doesn't curate.