John Mueller, Google's Senior Search Analyst, told the Google podcast Search Off the Record that llms.txt files cannot be used by large language model systems to differentiate which website to surface, saying "It's basically you're telling these systems, like, I have the best website ever." (reported by Search Engine Journal and TechJuice). Mueller added a narrow use case: llms.txt may be helpful once an agent is already on a site to navigate or finish a task. Independent telemetry from Ahrefs supports low impact: Ahrefs analysed 137,000 domains and reported 97% of llms.txt files received zero requests in May 2026 and only 28% of domains published the file. Industry coverage notes Chrome Lighthouse added an llms.txt audit while Google documentation labeled such files nonessential for generative search (Ahrefs, SEJ).
What happened
John Mueller, Google's Senior Search Analyst, said on the Google podcast Search Off the Record that llms.txt files cannot be relied on by LLM systems to decide which website to surface, and that the files are self-reported and therefore not a differentiator, according to reporting by Search Engine Journal and TechJuice. Mueller was quoted saying, "It's basically you're telling these systems, like, I have the best website ever. And here are all of the pages that everyone must go to." He also described a narrow operational role: if an assistant has already landed on a site, an llms.txt file might help the agent find specific pages or complete a task, per TechJuice and SEJ.
What the data shows
Ahrefs analysed server logs from 137,000 domains and reported that 28% of those domains published an llms.txt file and 97% of the files received zero requests in May 2026, with most fetches coming from bots, Ahrefs reported. Ahrefs further found that requests that did occur were mostly from audit tools and non-AI crawlers, and that named AI retrieval bots accounted for a small share of activity, according to the Ahrefs writeup cited by Search Engine Journal.
Editorial analysis - technical context
Self-reported discovery signals face a classic signal-quality problem: when every site can publish identical promotional metadata, downstream models and rankers have limited ability to distinguish genuine authority from self-promotion. Industry observers have made the same comparison to the now-obsolete meta keywords tag; this comparison appears in multiple reports summarising Mueller's remarks (SEJ, TechJuice). From a systems perspective, agents that combine retrieval with model scoring rely on multiple independent signals for source selection, and a single site-provided manifest is a weak, easily gamed input.
Context and significance
Reporting also notes a broader ecosystem debate. Ahrefs highlighted that Chrome's Lighthouse shipped an llms.txt audit and Google's public guidance included a "mythbusting" note saying machine-readable files like llms.txt are not required to appear in generative AI search. The combination of official guidance plus empirical telemetry showing near-zero reads for most files suggests limited current utility for mainstream sites, while specialised developer or documentation sites and a small set of coding agents show some usage.
What to watch
Editorial analysis: observers should monitor three indicators:
- •adoption rates beyond technical audiences, measured by the percentage of domains publishing llms.txt
- •user-agent traffic to llms.txt paths, especially from major retrieval bots such as GPTBot and Perplexity's agents as tracked in server logs
- •whether major AI assistant platforms announce documented support or formal ingestion of llms.txt into their retrieval pipelines. Also watch tooling: audits (Lighthouse) and validators that could raise awareness and change how authors produce the file
Practical takeaway
Editorial analysis: for most sites, improving HTML structure, internal linking, and canonical content remains the observable, high-value work for discoverability by search engines and AI assistants. The current evidence base-Google commentary plus Ahrefs telemetry-does not support treating llms.txt as a substitute for those fundamentals.
Scoring Rationale #
Mueller confirming that llms.txt cannot guide LLM site selection is a useful practitioner signal, and Ahrefs server-log data (97% zero requests across 137K domains) gives it empirical weight. The story is relevant to engineers building RAG or agent-discovery integrations, but it is essentially an SEO/discoverability clarification rather than a frontier AI development, placing it in the solid mid-range.
Practice with real Ad Tech data
90 SQL & Python problems · 15 industry datasets
[Active Search Campaigns by BudgetEasy](/problems/sql/active-search-campaigns-by-budget)
[High CPC Clicks & Poor Landing PagesMedium](/problems/sql/high-cpc-clicks-poor-landing-page)
[Campaign ROAS by Attribution ModelHard](/problems/sql/campaign-roas-by-attribution-model)
250 free problems · No credit card