/llms.txt
file placed in a website's root directory that provides a condensed, LLM-friendly version of the site's content. It's basically the robots.txt
of the generative AI era, designed to give crawlers and agents a curated map of the most important documentation and context without them having to scrape thousands of messy HTML pages.The logic is sound. We've all seen LLMs hallucinate because they're pulling from a cached version of a website's "Terms of Service" instead of the actual API documentation. By providing a markdown-based summary file, a site owner could theoretically steer an LLM toward the truth.
How the proposed standard works #
If this actually catches on, the implementation is dead simple. You just drop a text file at the root of your domain.
> Short description of the site and its purpose.
## Core Documentation
- [Getting Started](/docs/start): How to install and run the tool.
- [API Reference](/docs/api): Full list of endpoints and parameters.
## Optional Context
- [Blog](/blog): Latest updates and tutorials.
The idea is that an AI agent, upon hitting a domain, would first check for this file to get a "cheat sheet" of the site's structure before diving into specific pages. It's essentially a manual override for the RAG (Retrieval-Augmented Generation) process, allowing the human author to define what is "high signal" and what is noise.
The adoption gap #
Here is the problem: no major AI platform has officially confirmed they use or prioritize this standard. OpenAI, Anthropic, and Google have their own proprietary crawling and indexing pipelines. They aren't necessarily looking for a "suggestion" file when they've already spent millions of dollars building sophisticated scrapers that can parse almost any DOM structure.
For this to become a real-world AI workflow standard, we need the LLM agents themselves—like Claude Code or various open-source autonomous agents—to treat /llms.txt
as the primary source of truth. If a developer is using a local LLM agent to debug a library, and that agent checks the llms.txt
file first, the efficiency gain is massive. It reduces token waste and prevents the model from getting lost in navigation menus or footer links.
Practical utility for devs #
Despite the lack of "official" backing from the giants, it's still a smart move for anyone maintaining a public API or a technical doc site. It's a low-effort, high-reward way to make your project more "agent-friendly." If you're building a tool and want to ensure that prompt engineering for your library is easier for others, giving the AI a curated map is the way to go.
It’s a bet on the future of the web. We're moving from a "human-readable" web to one where the primary consumers are headless browsers and LLM agents. Whether this specific file format wins or something else takes over, the need for a structured, AI-optimized index is undeniable.
OpenAI just killed off its catastrophic risk team 9h ago
AI is making every side project feel like meaningless slop 18h ago
Joining an AI startup while my social circle actively despises 21h ago
AI safety regulations are becoming a convenient shield for 22h ago
HarnessRouter makes it way easier to switch between Claude Code 23h ago
Dario Amodei thinks the AI backlash is actually a trust crisis 1d ago
Next Palantir is swallowing USA Today's data and the journalists are →