Part of the Agent Readiness course — the web standards that decide whether an AI agent can actually read, understand, and act on your site. Measure any page against these with the Core Agent Vitals analyzer.
What it is #
llms.txt
is a single markdown file you publish at your site root — https://your-site.com/llms.txt
. It gives LLMs and AI agents a curated, machine-readable map of your site: your name, a one-line description, and the handful of links that actually matter (docs, pricing, API, key articles), each with a short note on what it contains.
A companion file, llms-full.txt
, goes further: it inlines the actual content of those key pages as clean markdown, so an agent can answer questions about your site from one fetch — no crawling, no HTML parsing, no JavaScript execution.
Think of llms.txt
as the table of contents and llms-full.txt
as the printed book. The spec lives at llmstxt.org.
Why agents need it #
An AI agent works in a loop: observe → reason → act. The quality of everything downstream depends on that first observation. Without a machine-readable index, the agent's only option is to fetch your homepage HTML, strip the navigation and boilerplate, guess which links are relevant, fetch those, and repeat — spending tokens and time on markup it will throw away, and frequently landing on the wrong page.
llms.txt
collapses that whole discovery phase into one cheap, high-signal read:
Lower token cost. A 200-line markdown index costs a fraction of rendering and tokenizing full HTML pages. On the analyzer this shows up directly inToken Cost (TC).** Fewer wrong answers.You decide what the agent sees first, so it stops inferring your structure from a cluttered DOM. Deterministic entry points.**Agents that support the convention read/llms.txt
before crawling — you get to curate the first impression.
How to implement #
1. Create /llms.txt — an H1 with your name, a blockquote one-liner, then link sections:
> Developer documentation and API reference for the Acme platform.
## Docs
- [Quickstart](https://acme.com/docs/quickstart): Install and make your first call in 5 minutes.
- [Authentication](https://acme.com/docs/auth): API keys, OAuth, and scopes.
## API
- [REST reference](https://acme.com/api): All endpoints, params, and response shapes.
## Optional
- [Changelog](https://acme.com/changelog): Release notes.
2. (Recommended) Create /llms-full.txt — the same structure, but paste the clean markdown content of each key page inline. Generate it from your existing markdown/MDX at build time so it never drifts.
3. Serve both as text/plain at the root. On a static host, drop the files in
public/
. On a framework, add two routes that return the files with Content-Type: text/plain; charset=utf-8
.## Validate
curl -s https://your-site.com/llms.txt | head
You should get markdown, HTTP 200, and a text/plain
content type. Then run your site through the Core Agent Vitals analyzer — the Agent Discoverability panel checks for llms.txt
at both /llms.txt
and /.well-known/llms.txt
, and flags it if the H1/blockquote structure is missing.
Common mistakes #
Publishing it as HTML. If your framework wraps it in a layout, agents get a web page, not markdown. Serve rawtext/plain
.Listing every URL.llms.txt
is acuratedindex, not a sitemap. If it lists 400 links it's noise — link the 5–15 pages that matter and let the sitemap handle the long tail.No H1 or blockquote. Parsers key off the# Title
and> one-liner
. Skip them and tools treat the file as malformed — it fails silently.Letting If you hand-maintain the inlined content, it drifts from the real pages within weeks. Generate it from source at build time or don't ship it.llms-full.txt
rot.Blocking the crawlers that would read it. Anllms.txt
behind arobots.txt
disallow, a login wall, or a bot-challenge is invisible. Keep the file and the pages it points to publicly reachable.
Next in the course: Sitemaps for Agent Discovery — the table of contents for everything llms.txt doesn't curate.