According to an analysis published by Ahrefs, of 137,000 domains monitored, 28% published a llms.txt file and 97% of those files received zero requests in May 2026. Ahrefs reports that of roughly 38,000 valid files, only about 1,100 received any traffic and that 96% of requests that did occur came from bots. Ahrefs also reports 19.5% of fetches were from named AI tools while 12% came from audit and scanner tools. Search Engine Journal relays the same dataset and highlights a finer breakdown of AI-related user agents, noting tools such as GPTBot, Claude-Code, PerplexityBot, and Slackbot among the fetchers.
What happened
According to Ahrefs, it analyzed server logs and live traffic across 137,000 domains and found that 28% of those domains publish a llms.txt file and that 97% of published llms.txt files received no requests in May 2026. Ahrefs reports that of roughly 38,000 valid files, only about 1,100 received any traffic. Ahrefs also reports 96% of requests that reached llms.txt files were from bots, and that 19.5% of fetches came from named AI tools. Ahrefs states 12% of fetches originated from audit, scanner, or research tools. Search Engine Journal, reporting on the Ahrefs dataset, notes an alternate headline figure that AI retrieval bots accounted for 1.1% of llms.txt requests while also describing a breakdown where AI bot categories together made up about 19% of fetches and coding agents accounted for roughly 10% of those.
Technical details
Per Ahrefs, llms.txt is a single index file placed at a site root that summarizes a site for automated agents; Ahrefs traces the proposal back to a 2024 proposal from an Answer.AI / fast.ai co-founder. Ahrefs reports that named fetchers in the logs included GPTBot, Claude-Code, PerplexityBot, and non-AI agents such as Slackbot and standard web crawlers. Ahrefs also reports that requests to non-existent llms.txt paths returned 404s and drew no AI bot traffic in their dataset, and that the Chrome Lighthouse llms.txt audit produced roughly 1 in 1,000 fetches in the sample.
Industry context
The Ahrefs dataset documents very low live adoption and even lower active consumption of llms.txt across a broad sample of sites. A sizable share of observed fetches are coming from tooling and research activity rather than production AI retrieval systems, which suggests current agent behaviour does not yet rely on a widely published machine-readable index.
Context and significance
For practitioners building retrieval layers or advising site owners, the data indicates that publishing a llms.txt file has had limited observable payoff in May 2026. The dataset illustrates a common pattern when a new machine-readable web convention emerges: early attention from audits, validators, and researchers often outpaces real-world consumer or agent adoption.
What to watch
Observers should track three indicators: adoption rate changes in web analytics (the percentage of domains publishing llms.txt), the share of AI retrieval traffic in server logs versus audit/scanner traffic, and statements or telemetry from major AI agent operators about whether they will consult llms.txt at scale. Changes in Chrome tooling, major search or assistant vendor guidance, or clear increases in named-agent fetch rates would meaningfully shift the current picture.
Scoring Rationale #
Ahrefs provides the first large-scale traffic snapshot of llms.txt across 137,000 domains, offering useful baseline data for practitioners. However, near-zero adoption of the format by production AI agents in May 2026 reflects a single vendor's dataset at an early point in the standard's development, limiting its immediate practitioner impact.
Practice interview problems based on real data
1,500+ SQL & Python problems across 15 industry datasets — the exact type of data you work with.