{"slug": "we-put-llms-txt-on-83-websites-in-12-weeks-openai-s-crawler-read-it-7-times", "title": "We put llms.txt on 83 websites. In 12 weeks OpenAI's crawler read it 7 times", "summary": "EZY Research's 12-week study of 83 websites found OpenAI's GPTBot fetched llms.txt only 7 times versus 3,990 robots.txt fetches, while Meta's crawler fetched llms.txt 193 times, more than its 172 robots.txt fetches. PerplexityBot fetched llms.txt zero times despite 775 robots.txt fetches, and Googlebot fetched it 67 times against 5,125 robots.txt fetches. The study concludes robots.txt remains the primary file AI crawlers check, with llms.txt adoption by major engines still early.", "body_md": "[EZY Research](/research) · Study 1 · July 27, 2026\n\n# We put llms.txt on 83 websites. In 12 weeks, OpenAI's crawler read it 7 times.\n\n7\n\nOpenAI llms.txt fetches\n\nAcross 83 sites, 12 weeks\n\n3,990\n\nOpenAI robots.txt fetches\n\nSame sites, same window\n\n0\n\nPerplexity llms.txt fetches\n\nWhile fetching robots.txt 775 times\n\n[llms.txt](https://www.ezy.ai/blog/why-llms-txt-is-important) is the most talked-about file in AI search. Across our 556-domain research panel, 83 connected sites had llms.txt live with server logs we could watch, so we watched them for 12 weeks to see which AI crawlers actually fetch the file.\n\nAdoption of the standard is real and accelerating: Originality.ai counted llms.txt instances growing 8.8x in a year, to 36,120 files across the 3 million sites it scans, and publishers as large as TIME are building agent-readable versions of their content. What nobody had published is whether the crawlers those files are written for actually read them. Server logs answer that.\n\n## The counts, per crawler, same sites, same window\n\n| Crawler | robots.txt fetches | llms.txt fetches |\n|---|---|---|\n| OpenAI (GPTBot family) | 3,990 | 7 |\n| Anthropic (ClaudeBot) | 3,120 | 9 |\n| PerplexityBot | 775 | 0 |\n| Googlebot | 5,125 | 67 |\n| Meta-ExternalAgent | 172 | 193 |\n\nThe same crawlers that pulled robots.txt thousands of times pulled llms.txt in single digits. Perplexity fetched it zero times. The one major AI company reading llms.txt at scale right now is Meta, whose crawler fetched it more often than robots.txt.\n\nThe file does get read - just mostly by the wrong readers. Generic and unverified bots fetched it 1,001 times, and human visitors 564 times. And it gets found fast: for sites where anything fetched it, the median gap between deployment and first fetch was zero days. Discovery works. Adoption by the big engines is what's early.\n\nWhat this means in practice: robots.txt is where AI access is actually decided today. It's the file every major crawler checks constantly, and it's where sites accidentally block GPTBot and ClaudeBot without knowing. llms.txt is cheap, standards-track insurance - worth having on every site because Meta already reads it and the others may follow - but anyone selling it as the thing that gets you cited is selling ahead of the evidence.\n\nAt EZY we generate and upload both files in one click, and we built it in that order for exactly this reason: robots.txt is the door, llms.txt is the map, and the logs show which one the visitors are using. If you want to check what AI can see on your own site today, the free analysis takes a minute.\n\n## Methodology\n\nData: EZY.ai's 556-domain research panel - 125 connected business websites (WordPress and Cloudflare) with server-side logs, and 387 never-connected audited sites, 27 Apr-19 Jul 2026, UTC. Bots identified by user agent with signature verification where available. Panel skews toward small-business sites whose owners sought an AI-visibility tool. Aggregates only. Full tables available on request.\n\nAggregate CSVs for every table are available on request. The fixed-cohort check (42 domains present all quarter) is applied to all growth claims; raw-panel growth is reported only alongside it. Dataset license: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).\n\n## More from EZY Research\n\nFree AI Visibility Check\n\n## How visible is your website to AI?\n\nRun a free AI Visibility Check and discover what ChatGPT, Claude and other AI systems can find about your business.\n\n[Check my website free](https://www.ezy.ai/?utm_source=research&utm_medium=article&utm_campaign=free_ai_visibility_check)", "url": "https://wpnews.pro/news/we-put-llms-txt-on-83-websites-in-12-weeks-openai-s-crawler-read-it-7-times", "canonical_source": "https://www.ezy.ai/research/do-ai-bots-read-llms-txt", "published_at": "2026-08-13 02:01:33+00:00", "updated_at": "2026-08-13 02:10:41.593244+00:00", "lang": "en", "topics": ["ai-research", "ai-tools", "ai-infrastructure"], "entities": ["EZY Research", "OpenAI", "Anthropic", "Perplexity", "Google", "Meta", "Originality.ai", "TIME"], "alternates": {"html": "https://wpnews.pro/news/we-put-llms-txt-on-83-websites-in-12-weeks-openai-s-crawler-read-it-7-times", "markdown": "https://wpnews.pro/news/we-put-llms-txt-on-83-websites-in-12-weeks-openai-s-crawler-read-it-7-times.md", "text": "https://wpnews.pro/news/we-put-llms-txt-on-83-websites-in-12-weeks-openai-s-crawler-read-it-7-times.txt", "jsonld": "https://wpnews.pro/news/we-put-llms-txt-on-83-websites-in-12-weeks-openai-s-crawler-read-it-7-times.jsonld"}}