Llms.txt V2 Adds Formal Markdown Linking For AI Agents

Jeremy Howard, creator of llms.txt, released version 2 of the specification on August 10, adding formal Markdown linking for AI agents. The update introduces two URL patterns for Markdown pages and two link relations (rel="alternate" and rel="describedby") to help agents discover Markdown versions, based on two years of practical experience and adoption by platforms like Mintlify and companies like Anthropic, OpenAI, and Google's Gemini team. Google's Search team maintains that it does not use llms.txt files, while Chrome's Lighthouse includes an Agentic Browsing check for the file.

Jeremy Howard, the Answer.AI developer who created llms.txt, published a version 2 update https://llmstxt.org/ to the spec on August 10. It’s the first revision since the format launched, and it adds standard ways for agents to discover Markdown versions of pages. The 2024 proposal included the llms.txt file and Markdown versions of individual pages, which are published at the same URL with .md added to the filename. V2 keeps this but introduces a second URL pattern for Markdown pages and formal ways for agents to locate them. For sites already using the file, updating it to V2 requires only a minor change rather than a complete rebuild. What Changed In V2 V2 now supports two URL patterns for Markdown pages. Sites can either add .md to the full filename, which was the only pattern supported before, making /docs/tutorial.html become /docs/tutorial.html.md, or they can replace the extension altogether, turning the same page into /docs/tutorial.md. To make it easier for agents to discover Markdown pages, the V2 update introduces two link relations. A rel=”alternate” attribute with type=”text/markdown” provides a direct link from a page to its Markdown version. Additionally, a rel=”describedby” attribute points to the llms.txt file that covers it, because a single llms.txt file can describe all pages under its path. Sites can include link relations either as standard HTML link elements within the page head or through an HTTP Link response header. The header option is versatile, as it can be used on files that aren’t even HTML, like Markdown pages. Plus, it can be added via server or CDN setup, so there’s no need to change a site’s templates or code. Why Howard Made The Change Howard said the update is based on two years of practical experience. Now, thousands of sites are publishing an llms.txt file, and platforms like Mintlify help by creating one for every site they host. Google’s Chrome team has even added a check for this file to Lighthouse’s Agentic Browsing category, which we discussed when it was introduced https://www.searchenginejournal.com/googles-llms-txt-guidance-depends-on-which-product-you-ask/575431/ in May. Additionally, Anthropic https://docs.anthropic.com/llms.txt , OpenAI https://developers.openai.com/llms.txt , and Google’s Gemini team https://ai.google.dev/gemini-api/docs/llms.txt now publish their own llms.txt files for their developer documentation. As adoption grew, a real gap showed up. The llms.txt file pointed agents toward pages, but nothing in the spec told them where the Markdown versions actually lived. Where This Leaves Google’s Guidance The update doesn’t change the positions of Google’s two product teams. Google’s Search team has said for over a year that support for llms.txt isn’t on their agenda, and the company’s AI optimization guide https://developers.google.com/search/docs/fundamentals/ai-optimization-guide states that Google Search itself doesn’t utilize these files. The guide also notes that maintaining one of these files “will neither harm nor help your site’s visibility or rankings” because Google Search ignores them. Chrome’s Lighthouse tool has an Agentic Browsing check https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt that attempts to retrieve the llms.txt file and flags server errors. A missing file that returns a 404 is treated as N/A instead, because providing the file is optional. The tool views this as a way to help browser agents read a site’s layout more quickly. However, it doesn’t specifically test for new link relations or Markdown-page discovery. Why This Matters The update to V2 involves adding two link relations as HTML link elements or using one HTTP header. What’s really important is what this update indicates about who the format is designed to serve. Coding agents and document tools are already reading llms.txt files to help speed up API lookups. This use case isn’t related to whether Google uses the file as a ranking input. V2 clarifies this behavior rather than making a new case for search visibility. Websites that maintain an llms.txt file primarily to influence AI Overviews or AI Mode citations still have no data or Google statement backing that goal. Those who keep it because coding agents, IDE tools, or documentation platforms already use it now have a clearer spec to reference and a defined way to direct tools to Markdown content when available. Looking Ahead Howard’s changes page https://llmstxt.org/changes.html shows every addition to the llms.txt spec along with the reasoning behind it. V2 is still open for feedback on GitHub https://github.com/AnswerDotAI/llms-txt , so if you’re using the link relations now, remember that the syntax might still change before everything is finalized as a stable version. Featured Image: EyeFound/Shutterstock