On 7 September 2026 I fetched the robots.txt of the 5,000 highest ranked sites in the Tranco list. 2,771 of them served one. This page lists what those files say about each AI crawler, and the raw data is available as JSON. Shares below are of the 2,771 sites with a robots.txt.
Search crawlers. These decide whether you get cited. #
| Crawler | Company | What it does | Sites blocking it | Named in robots.txt | Share blocking |
|---|---|---|---|---|---|
| OAI-SearchBot | OpenAI | indexes pages for ChatGPT search | 241 | 334 | 8.7% |
| PerplexityBot | Perplexity | indexes pages for Perplexity answers | 359 | 427 | 13.0% |
| Claude-SearchBot | Anthropic | indexes pages for Claude's web search | 250 | 215 | 9.0% |
Live fetchers. These pull a page when a user asks about it. #
| Crawler | Company | What it does | Sites blocking it | Named in robots.txt | Share blocking |
|---|---|---|---|---|---|
| ChatGPT-User | OpenAI | fetches a page when a ChatGPT user asks about it | 303 | 388 | 10.9% |
| Perplexity-User | Perplexity | fetches a page when a Perplexity user asks about it | 259 | 204 | 9.3% |
| Claude-User | Anthropic | fetches a page when a Claude user asks about it | 253 | 215 | 9.1% |
Training crawlers. Blocking these does not affect citations. #
| Crawler | Company | Sites blocking it | Named in robots.txt | Share blocking |
|---|---|---|---|---|
| CCBot | Common Crawl | 573 | 538 | 20.7% |
| Bytespider | ByteDance | 542 | 463 | 19.6% |
| GPTBot | OpenAI | 535 | 630 | 19.3% |
| ClaudeBot | Anthropic | 516 | 552 | 18.6% |
| meta-externalagent | Meta | 476 | 408 | 17.2% |
| Google-Extended | 459 | 514 | 16.6% | |
| Amazonbot | Amazon | 428 | 383 | 15.4% |
| Applebot-Extended | Apple | 413 | 444 | 14.9% |
| Diffbot | Diffbot | 377 | 264 | 13.6% |
| cohere-ai | Cohere | 376 | 290 | 13.6% |
| omgili | Webz.io | 376 | 259 | 13.6% |
| TimpiBot | Timpi | 312 | 184 | 11.3% |
Classic search crawlers, for comparison #
| Crawler | Company | What it does | Sites blocking it | Named in robots.txt | Share blocking |
|---|---|---|---|---|---|
| Googlebot | Google search | 77 | 320 | 2.8% | |
| bingbot | Microsoft | Bing search, which ChatGPT search also draws on | 95 | 253 | 3.4% |
The mistake that costs citations #
535 sites block GPTBot, the crawler OpenAI uses to gather training data. 238 of those, 44.5%, also block OAI-SearchBot, the separate crawler that indexes pages for ChatGPT search. The other 297 block GPTBot alone, which keeps them out of training and still lets ChatGPT cite them. OpenAI documents the two crawlers separately, and most of these sites otherwise welcome search engines, so for many of the 238 this looks like an accident rather than a decision.
296 sites block at least one AI search crawler and still allow Googlebot. Among the largest: facebook.com, instagram.com, twitter.com, amazon.com, x.com, tiktok.com, pinterest.com, yahoo.com, ebay.com, imdb.com, unsplash.com, quora.com. News publishers on the list include nytimes.com, cnn.com, theguardian.com, forbes.com, bbc.com, reuters.com, washingtonpost.com, wsj.com, bloomberg.com, cnbc.com. Big publishers often do this on purpose, for licensing reasons. For a smaller business it is nearly always unintended. Check your own site if you are not sure which crawlers you block.
Can the crawlers read the page once they get in #
A separate check on 31 August 2026 fetched the homepage of 607 sites ranked around 10,000 in the same list, the way a crawler that does not run JavaScript sees them. 79.2% had substantial readable text. 16.0% were effectively empty without JavaScript, under 250 characters of text. 4.8% were thin, under 800 characters. 58.0% had no JSON-LD structured data and 26.4% had no meta description. So roughly one site in five that lets AI crawlers in still gives them nothing to read.
How many of the top 5,000 sites publish an llms.txt #
On 6 September 2026 I requested /llms.txt from the same 5,000 domains. 373 of them, 7.5%, served a file that follows the proposed format (a markdown H1 first, at least one link). 2201 answered 404. 180 returned an HTML page instead, which is what a site does when it serves its app for any path, and 347 redirected somewhere else. 122 returned text that was not an llms.txt. 416 refused the request outright and 1241 could not be reached at all; both are counted, not treated as absent.
| Tranco rank | Sites with a valid llms.txt | Sites checked | Share |
|---|---|---|---|
| Ranks 1 to 100 | 11 | 100 | 11.0% |
| Ranks 101 to 1,000 | 75 | 900 | 8.3% |
| Ranks 1,001 to 5,000 | 287 | 3999 | 7.2% |
55 of the 373 also publish an llms-full.txt, and 320 include the blockquote summary the format asks for. The highest ranked sites with one: cloudflare.com, azure.com, github.com, fastly.net, digicert.com, wordpress.org, adobe.com, opera.com, samsung.com, sentry.io, wordpress.com, unity3d.com. Every domain with a valid file is listed in the JSON below. For the question people search, whether you need an llms.txt to show up in ChatGPT: adoption is 7.5% across the 5,000 and 11.0% in the top 100, and none of the AI search engines has documented reading it for search, so treat it as cheap and optional. The generator and validator will make one if you want it.
Method #
Domains come from the Tranco top list, a research ranking that averages several traffic lists. Each robots.txt was fetched once with my own identified user agent. I never impersonate another crawler. A crawler counts as blocked when the most specific robots.txt group that applies to it disallows the site root. If no group names the crawler, the catch-all group applies, so a site that disallows everyone counts as blocking every crawler. The "named in robots.txt" column counts files that mention the crawler at all, whether to allow or block it.
Percentages use the 2,771 sites that served a robots.txt as the base. Sites that returned an error, a redirect to HTML, or an empty file are excluded. The rendering check was a single homepage fetch per site with scripts and styles stripped, so a site that serves crawlers a different page than browsers can be misjudged. Everything here is a snapshot. robots.txt files change, and I rerun the census and update this page.
Get the data #
The per-crawler counts, the misconfiguration counts, the named examples, the rendering results and the llms.txt adoption list are at /data/census.json. The data is free to reuse under CC BY 4.0. A link back to this page is the only ask.