Everyone has an opinion on whether websites are blocking AI crawlers. We wanted the numbers, so on 4 November 2026 we downloaded the Tranco ranking of popular domains, took the top 2,000, fetched each site's robots.txt, and parsed the 1,069 that gave us a usable file for rules about 13 AI crawlers. We also checked whether each site serves an llms.txt. The per-site results are in a CSV you can download.
Sites blocking each training crawler site-wide (n = 1,069)
| Crawler | Operator | Sites blocking | Share |
|---|---|---|---|
| CCBot | Common Crawl | 221 | 20.7% |
| Bytespider | ByteDance | 214 | 20.0% |
| GPTBot | OpenAI | 211 | 19.7% |
| ClaudeBot | Anthropic | 201 | 18.8% |
| Meta-ExternalAgent | Meta | 172 | 16.1% |
| Google-Extended | 170 | 15.9% | |
| Applebot-Extended | Apple | 162 | 15.2% |
These are the crawlers vendors say collect pages for model training. OpenAI says disallowing GPTBot indicates your content should not be used for training, and Anthropic says the same of ClaudeBot. Blocking tends to come as a package: of the 268 sites that block at least one of the six major training crawlers, 87% block two or more and 74% block three or more. That looks like a policy decision, not a one-off rule.
Sites blocking AI search crawlers and user-triggered fetchers site-wide (n = 1,069)
| Crawler | Job | Sites blocking | Share |
|---|---|---|---|
| PerplexityBot | Builds Perplexity's search index | 124 | 11.6% |
| Claude-SearchBot | Improves Claude's search results | 87 | 8.1% |
| OAI-SearchBot | Surfaces sites in ChatGPT search | 72 | 6.7% |
| ChatGPT-User | Fetches a page when a ChatGPT user asks | 101 | 9.4% |
| Perplexity-User | Fetches a page when a Perplexity user asks | 96 | 9.0% |
| Claude-User | Fetches a page when a Claude user asks | 89 | 8.3% |
Overall, 131 sites (12.3%) block at least one of the three search crawlers, and 63 sites (5.9%) block all three. Those are the sites that have chosen to be invisible to AI answers, not just to AI training. Keep in mind that the user-triggered fetchers are a special case: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Blocking those two in robots.txt may do less than it appears to.
This is the most useful finding for anyone writing a robots.txt. Of the 211 sites that block GPTBot, 139 (65.9%) do not block OAI-SearchBot. That matches how OpenAI documents the two: the settings are independent, so a site can opt out of training while staying eligible for ChatGPT search. The remaining 72 sites block both, and OpenAI says a site that opts out of OAI-SearchBot will not appear in ChatGPT search answers, though it can still show up as a navigational link.
If you want to be cited, that split is the policy to copy: block the training crawlers if you object to training, and leave the search and user agents alone. Our guide to which AI crawlers to allow has an example file.
Thirteen sites (1.2%) block GPTBot only because of a wildcard: a User-agent: * group with Disallow: /, alongside separate groups that let in search engines such as Googlebot. That is a legitimate way to write a strict policy, but it silently shuts out every AI crawler the author did not think to name, including the ones that drive citations. If your robots.txt is an allowlist, check that it names OAI-SearchBot, PerplexityBot and Claude-SearchBot on purpose.
Blocking by Tranco rank band
| Rank band | Sites analysed | Block GPTBot | Block any search crawler |
|---|---|---|---|
| 1-100 | 45 | 22.2% | 24.4% |
| 101-500 | 176 | 18.2% | 13.6% |
| 501-1,000 | 256 | 19.1% | 13.7% |
| 1,001-2,000 | 592 | 20.3% | 10.3% |
GPTBot blocking sits between 18% and 22% in every band. The one visible trend is that the very largest sites are likelier to block search crawlers too (24.4% in the top 100, against 10.3% in the 1,001-2,000 band), but only 45 sites in the top band had a readable robots.txt, so treat that as a hint, not a result. The sites blocking GPTBot include familiar names such as nytimes.com, bbc.com, reuters.com, cnn.com, amazon.com and medium.com.
77 sites (7.2%) already use a Content-Signal line in robots.txt. It comes from Cloudflare's Content Signals Policy and lets a site say, per use, whether it permits search, ai-input (answers and grounding) and ai-train. Cloudflare describes the signals as expressing preferences, not as technical countermeasures, and notes some companies may ignore them. We did not count Content-Signal lines as blocks, because they do not instruct a crawler to stay away.
177 of the 1,069 sites (16.6%) serve an llms.txt. Adoption was highest in the 101-500 band (23.9%) and lowest in the 1,001-2,000 band (12.3%). Sites with an llms.txt block GPTBot far less often than the rest (4.5% against 22.8%), which fits the file being adopted mostly by developer-tools, SaaS and documentation sites that want AI visibility. That is selection, not proof that llms.txt works. In fact, an Ahrefs study of 137,210 domains found that 97% of llms.txt files received no requests in May 2026. We cover this in llms.txt explained.
Share of sites blocking each crawler, by study. Samples and definitions differ, so compare patterns, not decimals.
| Study and sample | GPTBot | ClaudeBot | PerplexityBot | OAI-SearchBot |
|---|---|---|---|---|
| This study: 1,069 sites from the Tranco top 2,000, Nov 2026 | 19.7% | 18.8% | 11.6% | 6.7% |
| [AICrawlerCheck](https://aicrawlercheck.com/blog/ai-crawler-robots-txt-study-2026) : 4,962 robots.txt files from the Tranco top 10,000, Oct 2026 | 14.3% | 13.4% | 10.5% | 7.4% |
| [Suff Digital](https://www.suffdigital.com/resources/data-studies/ai-crawler-blocking-rate) , top 1,000 by Chrome UX Report rank, Sept 2026 | 23.1% | 21.6% | 14.1% | 12.6%* |
| [Suff Digital](https://www.suffdigital.com/resources/data-studies/ai-crawler-blocking-rate) , all 15.47M HTTP Archive sites, Sept 2026 | 8.7% | 8.3% | 1.7% | 2.1%* |
*Suff Digital reports OAI-SearchBot and ChatGPT-User as one figure, and counts a site as blocking when a robots.txt rule addresses that bot by name, so wildcard-only blocks are treated differently from our method. Three patterns hold across these studies: training crawlers are blocked more than search crawlers, popular sites block far more than the web at large, and many GPTBot blockers still allow ChatGPT search. On that last point the size varies: 65.9% in our sample, and about half (347 of 710) in AICrawlerCheck's. The safe summary is one half to two thirds.
User-agent: *, add the search crawlers explicitly.* group otherwise, and counts as blocked only when the result is a whole-site Disallow: / with no allow for the root. Partial blocks such as Disallow: /private/ do not count. Python's standard-library parser disagreed with ours on about 2% of sites, mostly because it ends a group at a blank line, which RFC 9309 does not./llms.txt returned HTTP 200, was not HTML, and began with a markdown heading. 96 sites returned an HTML page with a 200 status and were not counted.
What percentage of websites block GPTBot?
Among the 1,069 popular sites we could analyse, 19.7% block GPTBot site-wide in robots.txt. That is a sample of popular sites, not the whole web, and the share for small sites is probably lower.
Does blocking GPTBot remove my site from ChatGPT?
No. According to OpenAI's documentation, GPTBot controls use of your content for training, while OAI-SearchBot controls whether you appear in ChatGPT search. The settings are independent, and 65.9% of the sites that block GPTBot in our data still allow OAI-SearchBot.
Should I block AI crawlers?
Block the training crawlers if you object to your content being used for training. If you want AI answers to cite you, do not block the search and user-triggered crawlers. See which AI crawlers to allow.
Can I get the data?
Yes. The per-site results are in [this CSV](https://citedlens.com/data/ai-crawler-robots-study-2026-11.csv), with one row per site and a 0 or 1 for each crawler.
*Originally published at [Cited Lens](https://citedlens.com/blog/ai-crawler-blocking-study). You can check which AI crawlers your own robots.txt blocks with the [free audit](https://citedlens.com).*