EZY Research · Study 1 · July 27, 2026 7
OpenAI llms.txt fetches
Across 83 sites, 12 weeks
3,990
OpenAI robots.txt fetches
Same sites, same window
0
Perplexity llms.txt fetches
While fetching robots.txt 775 times llms.txt is the most talked-about file in AI search. Across our 556-domain research panel, 83 connected sites had llms.txt live with server logs we could watch, so we watched them for 12 weeks to see which AI crawlers actually fetch the file.
Adoption of the standard is real and accelerating: Originality.ai counted llms.txt instances growing 8.8x in a year, to 36,120 files across the 3 million sites it scans, and publishers as large as TIME are building agent-readable versions of their content. What nobody had published is whether the crawlers those files are written for actually read them. Server logs answer that.
The counts, per crawler, same sites, same window #
| Crawler | robots.txt fetches | llms.txt fetches |
|---|---|---|
| OpenAI (GPTBot family) | 3,990 | 7 |
| Anthropic (ClaudeBot) | 3,120 | 9 |
| PerplexityBot | 775 | 0 | | Googlebot | 5,125 | 67 | | Meta-ExternalAgent | 172 | 193 |
The same crawlers that pulled robots.txt thousands of times pulled llms.txt in single digits. Perplexity fetched it zero times. The one major AI company reading llms.txt at scale right now is Meta, whose crawler fetched it more often than robots.txt.
The file does get read - just mostly by the wrong readers. Generic and unverified bots fetched it 1,001 times, and human visitors 564 times. And it gets found fast: for sites where anything fetched it, the median gap between deployment and first fetch was zero days. Discovery works. Adoption by the big engines is what's early.
What this means in practice: robots.txt is where AI access is actually decided today. It's the file every major crawler checks constantly, and it's where sites accidentally block GPTBot and ClaudeBot without knowing. llms.txt is cheap, standards-track insurance - worth having on every site because Meta already reads it and the others may follow - but anyone selling it as the thing that gets you cited is selling ahead of the evidence.
At EZY we generate and upload both files in one click, and we built it in that order for exactly this reason: robots.txt is the door, llms.txt is the map, and the logs show which one the visitors are using. If you want to check what AI can see on your own site today, the free analysis takes a minute.
Methodology #
Data: EZY.ai's 556-domain research panel - 125 connected business websites (WordPress and Cloudflare) with server-side logs, and 387 never-connected audited sites, 27 Apr-19 Jul 2026, UTC. Bots identified by user agent with signature verification where available. Panel skews toward small-business sites whose owners sought an AI-visibility tool. Aggregates only. Full tables available on request.
Aggregate CSVs for every table are available on request. The fixed-cohort check (42 domains present all quarter) is applied to all growth claims; raw-panel growth is reported only alongside it. Dataset license: CC BY 4.0.
More from EZY Research #
Free AI Visibility Check
How visible is your website to AI? #
Run a free AI Visibility Check and discover what ChatGPT, Claude and other AI systems can find about your business.