# Ahrefs Finds 97% of llms.txt Files Receive No Requests > Source: > Published: 2026-06-16 05:20:10.017234+00:00 # Ahrefs Finds 97% of llms.txt Files Receive No Requests According to an analysis published by Ahrefs, of **137,000** domains monitored, **28%** published a llms.txt file and **97%** of those files received zero requests in May 2026. Ahrefs reports that of roughly **38,000** valid files, only about **1,100** received any traffic and that **96%** of requests that did occur came from bots. Ahrefs also reports **19.5%** of fetches were from named AI tools while **12%** came from audit and scanner tools. Search Engine Journal relays the same dataset and highlights a finer breakdown of AI-related user agents, noting tools such as GPTBot, Claude-Code, PerplexityBot, and Slackbot among the fetchers. ### What happened According to Ahrefs, it analyzed server logs and live traffic across **137,000** domains and found that **28%** of those domains publish a llms.txt file and that **97%** of published llms.txt files received no requests in May 2026. Ahrefs reports that of roughly **38,000** valid files, only about **1,100** received any traffic. Ahrefs also reports **96%** of requests that reached llms.txt files were from bots, and that **19.5%** of fetches came from named AI tools. Ahrefs states **12%** of fetches originated from audit, scanner, or research tools. Search Engine Journal, reporting on the Ahrefs dataset, notes an alternate headline figure that AI retrieval bots accounted for **1.1%** of llms.txt requests while also describing a breakdown where AI bot categories together made up about **19%** of fetches and coding agents accounted for roughly **10%** of those. ### Technical details Per Ahrefs, llms.txt is a single index file placed at a site root that summarizes a site for automated agents; Ahrefs traces the proposal back to a 2024 proposal from an Answer.AI / fast.ai co-founder. Ahrefs reports that named fetchers in the logs included GPTBot, Claude-Code, PerplexityBot, and non-AI agents such as Slackbot and standard web crawlers. Ahrefs also reports that requests to non-existent llms.txt paths returned 404s and drew no AI bot traffic in their dataset, and that the Chrome Lighthouse llms.txt audit produced roughly 1 in 1,000 fetches in the sample. ### Industry context The Ahrefs dataset documents very low live adoption and even lower active consumption of llms.txt across a broad sample of sites. A sizable share of observed fetches are coming from tooling and research activity rather than production AI retrieval systems, which suggests current agent behaviour does not yet rely on a widely published machine-readable index. ### Context and significance For practitioners building retrieval layers or advising site owners, the data indicates that publishing a llms.txt file has had limited observable payoff in May 2026. The dataset illustrates a common pattern when a new machine-readable web convention emerges: early attention from audits, validators, and researchers often outpaces real-world consumer or agent adoption. ### What to watch Observers should track three indicators: adoption rate changes in web analytics (the percentage of domains publishing llms.txt), the share of AI retrieval traffic in server logs versus audit/scanner traffic, and statements or telemetry from major AI agent operators about whether they will consult llms.txt at scale. Changes in Chrome tooling, major search or assistant vendor guidance, or clear increases in named-agent fetch rates would meaningfully shift the current picture. ## Scoring Rationale Ahrefs provides the first large-scale traffic snapshot of `llms.txt` across 137,000 domains, offering useful baseline data for practitioners. However, near-zero adoption of the format by production AI agents in May 2026 reflects a single vendor's dataset at an early point in the standard's development, limiting its immediate practitioner impact. Practice interview problems based on real data 1,500+ SQL & Python problems across 15 industry datasets — the exact type of data you work with. [Try 250 free problems](/problems)