Plenty of sites now tell AI companies not to train on them. Fewer seem to have noticed that the same companies run other bots too: the ones that search and fetch pages when someone asks ChatGPT or Claude a question, and cite them. Block the search ones and you’re out of the answer, not just the training set. So we asked a narrow question: of the sites that keep a company’s training bot out, how many keep its answer bots out as well?
For OpenAI: of the sites that block GPTBot by name, 17.1% also block OAI-SearchBot, ChatGPT’s search crawler. That’s the number we nearly published. Reading the files by hand, we kept meeting the same one, and it turned out to be two very different groups averaged together.
| Sites that block GPTBot by name | Sites | Also block OAI-SearchBot | Also block ChatGPT-User |
|---|---|---|---|
| With Cloudflare’s managed robots.txt | 609 | 0.3% | 1.0% |
| With rules they wrote themselves | 722 | 31.3% | 47.0% |
| All | 1,331 | 17.1% | 25.9% |
Nearly half the sites blocking GPTBot did it with one setting. Cloudflare offers a managed robots.txt that blocks a list of training bots and names no answer bot, so those stay allowed. Sites that wrote their own rules are a different story: about a third block ChatGPT’s search, and nearly half block the fetches ChatGPT makes when a person asks it to open a page. It isn’t a blanket ban, either. Only 7 of those 722 keep everyone out, Googlebot included, and 29.0% of the rest block OAI-SearchBot by name.
Every bot, every site #
Share of the 13,938 sites whose robots.txt we could read that keep each bot out of the whole site:
| Bot | Company | What it’s for | Whole site blocked |
|---|---|---|---|
| Googlebot | Search, and AI Overviews | 1.4% | |
| Bingbot | Microsoft | Bing, and Copilot | 1.8% |
| OAI-SearchBot | OpenAI | ChatGPT search | 3.5% |
| ChatGPT-User | OpenAI | pages ChatGPT opens for a person | 4.4% |
| Claude-SearchBot | Anthropic | Claude search | 3.2% |
| Claude-User | Anthropic | pages Claude opens for a person | 3.1% |
| PerplexityBot | Perplexity | Perplexity search | 4.6% |
| Perplexity-User | Perplexity | pages Perplexity opens for a person | 3.2% |
| GPTBot | OpenAI | training | 11.4% |
| ClaudeBot | Anthropic | training | 10.4% |
| Google-Extended | training Gemini | 9.5% | |
| Applebot-Extended | Apple | training | 9.1% |
| CCBot | Common Crawl | the open crawl many models train on | 11.7% |
| Meta-ExternalAgent | Meta | training | 9.7% |
Training bots are blocked about three times as often as answer bots. Search engines, almost never.
The same question for each company #
Of the sites that block the first bot by name:
| Blocks by name | Sites | Cloudflare managed | Also block | All of them | Own rules only |
|---|---|---|---|---|---|
| GPTBot | 1,331 | 45.8% | OAI-SearchBot | 17.1% | 31.3% |
| ChatGPT-User | 25.9% | 47.0% | |||
| ClaudeBot | 1,167 | 52.1% | Claude-SearchBot | 14.2% | 29.2% |
| Claude-User | 12.9% | 27.0% | |||
| Google-Extended | 1,056 | 57.4% | Googlebot | 0.6% | 1.1% |
| PerplexityBot | 383 | 1.8% | Perplexity-User | 43.1% | 43.9% |
Anthropic looks like OpenAI once the managed file is set aside. Google-Extended is the clean case: it only controls Gemini training, Googlebot does the search, and almost nobody who opts out of the first touches the second. Perplexity runs the other way. PerplexityBot is its search crawler, not a training one, so Cloudflare’s file doesn’t name it, and four in ten sites that block it also block the fetches people ask for.
Big sites and small ones #
GPTBot blocked by name, then OAI-SearchBot as well:
| Sample | Block GPTBot by name | Of them, Cloudflare managed | Also block OAI-SearchBot | Own rules only |
|---|---|---|---|---|
| Top 10,000 | 652 | 22.7% | 27.1% | 34.9% |
| Ranks 100,000 to 1,000,000 (10,000 drawn) | 416 | 62.7% | 9.1% | 24.5% |
| Every .ro domain in the list (4,419) | 263 | 76.0% | 4.9% | 19.0% |
Most of the gap between big sites and small ones is who uses the managed file. The long tail and the Romanian sites block GPTBot mostly through Cloudflare. Big sites mostly write their own rules, and when they do, they’re more likely to shut the answer bots out too.
How we did it #
- The sites. TheTranco list of 13 October 2026 (N26LW): the top 10,000, 10,000 drawn at random from ranks 100,000 to 1,000,000, and every .ro domain in it. 24,419 domains.
- The files. We didn’t ask the sites. Every robots.txt here is the copy Common Crawl fetched for itsSeptember 2026 crawl (CC-MAIN-2026-39), read from its archive. 16,631 domains had a copy; the other 7,788 the crawl never asked for. Redirects were followed through the archive, up to five hops, as Google does.
- What counts. A bot is blocked when its rules keep it off an ordinary page of the site, not just the home page. By name means the file has a group for that bot; a bot kept out only through User-agent: * isn’t counted as named. Files that keep Googlebot and Bingbot out too are blanket blocks, not an AI choice, and we count them apart.
- The parser. The open-sourcerobots-parser library, the one ourrobots.txt validator and our site audit read with, so all three read a file the same way.
Of the 24,419: 13,938 readable (1,683 of them with no robots.txt, which means everything is allowed), 1,360 refused the crawl a copy (782 of them with a bot challenge), 1,333 errors, 7,788 not in the archive.
Checking our own numbers #
We read 50 files by hand, drawn at random in four groups, without looking at what the code had said. On what each file was (a robots.txt, no file, a bot wall), 50 of 50 agreed. On each bot’s verdict, 1,676 of 1,680. The other four were two files with one pattern:
User-agent: PerplexityBot
Crawl-delay: 600
User-agent: Timpibot
Disallow: /
We read that as PerplexityBot slowed down, which is clearly what the author meant. Google’s own parser reads it as PerplexityBot blocked: it ignores the line it doesn’t know, and the two user agents run together. We judged every file the Google way as well. 63 of 12,255 change; the share that blocks each bot moves by 0.3 points at most, in any sample, and the 17.1% stays 17.1%. Only the smallest groups move more: 24 .ro sites block PerplexityBot by name, so one site there is four points.
Following redirects mattered as little. It made 542 more sites readable, moved the share that blocks each bot by 0.3 points at most, and left the 17.1% where it was.
What this can’t tell you #
The obvious objection first: these are Common Crawl’s copies from September, not what sites serve today to GPTBot. Before switching to the archive we’d read 192 of the same sites live. 92% gave the same verdict for every bot. Every difference was on a training bot, almost always “the archive’s copy blocks it, the live file doesn’t”, and most were Cloudflare’s managed file showing up in one and not the other. So the training shares may run a few points high; the answer-bot shares didn’t move.
- A robots.txt is a request, not a lock. A site can let a bot in on paper and block it at the firewall, or the reverse. Cloudflare’s AI bot settings work at the edge and never show up in the file; our post on them says what each one does.
- For two of the answer bots the request may not even be read. ChatGPT-User and Perplexity-User open a page because a person asked; OpenAI says robots.txt rules “may not apply” to the first (our page on it ), Perplexity that the second “generally ignores” them (ours ). For those two, this study counts what sites ask, not what happens.
- Sites that block Common Crawl’s crawler at the edge refused it a copy, so they’re in the refused count, not in the shares.
- Tranco ranks domains, not sites, and the .ro sample is whatever .ro domains it happens to rank.
The data and the code #
On Zenodo, doi.org/10.5281/zenodo.23416952: totals for every sample, the question for every sample, verdicts site by site for the top 10,000 (as our bot page promised, nothing site by site below that), and the scripts that select, read and judge the files. Run in order, they rebuild the three tables byte for byte from Common Crawl’s archive. The data is CC BY 4.0, the code MIT. To have a domain taken out of the data, write to bot@trakis.io.
If it’s your robots.txt #
If you want out of training but into the answers, check the answer bots by name: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User. Our robots.txt validator shows what your file asks of each one, and can rewrite it to keep training out and answers in. The AI crawlers guide says which bot is which.
Sources #
- Tranco, list N26LW , 13 October 2026.
- Common Crawl, CC-MAIN-2026-39 , September 2026: the columnar index, subset robotstxt, and the WARC files.
- Google Search Central, How Google interprets the robots.txt specification .
- IETF, RFC 9309, Robots Exclusion Protocol .
- Cloudflare, Managed robots.txt .
- robots-parser 3.0.1.
Last changed 4 November 2026.