When the llms.txt proposal emerged as a standard to help LLMs and autonomous agents navigate web documentation efficiently, it promised an alternative to messy HTML scraping.
In August 2026, the specification was formally updated to v2, introducing two major changes:
<link rel="describedby"> tags.
Our research team at GrowNexus audited 25 major SaaS platforms to measure how the industry is actually adopting the standard.
We examined 25 prominent platforms across developer tooling, CMS, infrastructure, and productivity software:
llms.txt file.rel="describedby"):
The most significant operational issue discovered is using llms.txt as a raw sitemap dump.
When autonomous agents (like Claude Code, Codex, or Hermes) parse context, scratchpads are token-budgeted. Files over 50KB consume 15,000+ tokens before the agent executes a single tool call, resulting in aggressive truncation.
A well-architected llms.txt should act as a curated index of core endpoints—not a full site mirror.
We have released the complete dataset and analysis scripts under the CC BY 4.0 license:
How is your engineering team approaching llms.txt implementation for your APIs?