{"slug": "llms-txt-v2-what-an-agent-actually-reads", "title": "llms.txt v2: what an agent actually reads", "summary": "A developer tested the August 2026 revision of the llms.txt proposal, running this site's own /llms.txt through the proposal's roughly twenty-line reference parser (llms_txt/miniparse.py) and finding that it crashed. The v2 changes define how agents discover the file: hierarchical /docs/llms.txt scoping with most-specific-file precedence, .md and .html.md Markdown variants, and rel=\"alternate\" type=\"text/markdown\" plus rel=\"describedby\" link relations delivered via HTML head or HTTP Link headers, while ## Optional loses its special meaning. The note includes a JavaScript port of the parser and helper functions for resolving covering llms.txt files and Markdown URLs.", "body_md": "**This note has an interactive demo.** [Try it in the original](https://richi.codes/article/2026-10-23-llms-txt-v2-what-an-agent-actually-reads)\n\nThe llms.txt proposal got its first big revision in August 2026, and the changes answer the obvious question nobody could answer before: given a page, how does an agent find the llms.txt that describes it? I read the new text, ran this site's own `/llms.txt` through the small reference parser that lives in the proposal's repository, and it crashed. The demo above shows both things: what a parser really gets out of the file, and how an agent walks from a page to its llms.txt and its Markdown version.\n\nIt's a Markdown file that gives agents a curated entry point into a site: a name, a short summary, some context and lists of links to LLM-friendly content. Jeremy Howard published the original in September 2024. It's still a proposal \"open for community input\", not a W3C or IETF standard. The text is clear about what it's for: robots.txt says what automated access is acceptable, while llms.txt \"is instead used on demand, when an agent needs information about a topic while assisting a user\". It's meant for inference, not training.\n\nThe Changes page lists four things:\n\n`/docs/llms.txt` covers everything under `/docs/`, and \"where more than one file applies, agents should use the most specific one\". This is what lets a GitHub Pages project site, which can't touch the host root, take part.`.md` appended (`page.html.md`). v2 also allows replacing the extension (` page.md`). URLs without a file name use `index.html.md` or `index.md`.` rel=\"alternate\" type=\"text/markdown\"` points to a page's Markdown version and `rel=\"describedby\"` points to the llms.txt that covers it, either as `<link>` elements or as an HTTP header.`## Optional` lost its special meaning.`Optional` is now just a convention for secondary links.\nThe header form is the interesting one, because you can add it in your server or CDN config without touching a single page, and it also works on non-HTML files. This is the proposal's own example:\n\n```\nLink: </docs/page.html.md>; rel=\"alternate\"; type=\"text/markdown\", </docs/llms.txt>; rel=\"describedby\"\n```\n\nAnd the equivalent in the page's `<head>`:\n\n```\n<link rel=\"alternate\" type=\"text/markdown\" href=\"https://dev.to/docs/page.html.md\">\n<link rel=\"describedby\" href=\"https://dev.to/docs/llms.txt\">\n```\n\nThe \"How it gets found\" tab in the demo builds that header for each page of a small example site. These are the two functions behind it, which I ran in Node with the outputs shown:\n\n```\n// The llms.txt files that cover a path, most specific first.\nfunction coveringLlms(path, files) {\n  return files\n    .filter(f => path.startsWith(f.slice(0, f.lastIndexOf('/') + 1)))\n    .sort((a, b) => b.length - a.length);\n}\n\n// page.html -> page.html.md ('append') or page.md ('replace').\nfunction markdownUrl(path, mode) {\n  if (path.endsWith('/')) return path + (mode === 'append' ? 'index.html.md' : 'index.md');\n  return mode === 'append' ? path + '.md' : path.replace(/\\.[^./]+$/, '') + '.md';\n}\n\nconst files = ['/llms.txt', '/docs/llms.txt'];\ncoveringLlms('/docs/api/fetch.html', files); // ['/docs/llms.txt', '/llms.txt']\nmarkdownUrl('/docs/api/fetch.html', 'replace'); // '/docs/api/fetch.md'\nmarkdownUrl('/docs/', 'append');                // '/docs/index.html.md'\n```\n\nUntick \"/docs/llms.txt exists\" and the docs pages fall back to the root file.\n\nThe proposal says the file is \"in a precise format allowing fixed processing methods (i.e. classical programming techniques such as parsers and regex)\". In order: an H1 with the name (the only required part), a blockquote with a short summary, zero or more paragraphs or lists with no headings, then H2 sections whose lists hold `[name](url)` links, optionally followed by `:` and notes.\n\nThe repository still ships a parser of about twenty lines, `llms_txt/miniparse.py`, and the demo's first tab is a line-for-line JavaScript port of it. I ran both on the same seven files, this site's llms.txt and the proposal's own included, and they return the same result in every case. Here is what I found by feeding it real input:\n\n``` python\nfrom miniparse import parse_llms_txt\n\nparse_llms_txt(\"# My site\\n\\n> What it is.\\n\\nOne line of details.\\n\\n## Docs\\n\\n- [Guide](https://example.com/guide.md): start here\\n\")\n# {'title': 'My site', 'summary': 'What it is.', 'info': 'One line of details.',\n#  'sections': {'Docs': [{'title': 'Guide', 'url': 'https://example.com/guide.md', 'desc': 'start here'}]}}\n```\n\n`[name](url)` link.`- GitHub: https://github.com/me` raises `AttributeError`. So does a long list item wrapped onto a second, indented line, because the parser splits sections line by line. This site's `/llms.txt` does both, which is why the \"richi.codes today\" example opens in red.`>` lines keeps its first line as the summary and dumps the rest into the details.`##`, the summary comes back as `None` and the blockquote ends up in `info`. With only an H1 before the first `llms.txt` in the proposal's own repository comes back with `summary: None` for this reason.`###` also splits.`^##`, so `### Getting started` becomes a section called `# Getting started`. The spec doesn't allow headings before the first H2 anyway.\nAn agent reading the file as plain Markdown won't care about any of this. But if a tool parses llms.txt the way the proposal invites it to, these details decide whether your links survive. The \"richi.codes fixed\" example shows the cheap fix: one line per link, real Markdown links, and a short paragraph after the summary.\n\n*Originally published at [richi.codes](https://richi.codes/article/2026-10-23-llms-txt-v2-what-an-agent-actually-reads).*", "url": "https://wpnews.pro/news/llms-txt-v2-what-an-agent-actually-reads", "canonical_source": "https://dev.to/richicodes/llmstxt-v2-what-an-agent-actually-reads-3c43", "published_at": "2026-10-24 07:30:39+00:00", "updated_at": "2026-10-24 07:43:55.743241+00:00", "lang": "en", "topics": ["agent-protocols", "ai-agents", "ai-crawlers", "developer-tools"], "entities": ["Jeremy Howard", "llms_txt/miniparse.py", "GitHub Pages", "W3C", "IETF"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llms-txt-v2-what-an-agent-actually-reads", "markdown": "https://wpnews.pro/news/llms-txt-v2-what-an-agent-actually-reads.md", "text": "https://wpnews.pro/news/llms-txt-v2-what-an-agent-actually-reads.txt", "jsonld": "https://wpnews.pro/news/llms-txt-v2-what-an-agent-actually-reads.jsonld"}}