block AI crawlers?
We checked the robots.txt rules of 945 well-known websites for 14 AI crawlers: news, retail, SaaS, finance, government and more. October 2026 check, finished 22 October 2026. We check again every month.
If a site is not one of the 945 in the study, we check it live. 25.9%
block at least one AI crawler in robots.txt (193 of 746 sites we could read)
9.2%
block AI training but leave every AI search crawler open (69 sites)
13.9%
turned away our test request identifying as GPTBot or ClaudeBot (see the limits below)
Blocked most often #
14 crawlers | Crawler | Kind | Sites blocking it |
|---|---|---|
| [**CCBot**](https://getkavel.com/crawlers/ccbot/)Common Crawl | Training | 19.6% (146) |
| [**Bytespider**](https://getkavel.com/crawlers/bytespider/)ByteDance | Training | 18.4% (137) |
| [**ClaudeBot**](https://getkavel.com/crawlers/claudebot/)Anthropic | Training | 18.1% (135) |
| [**Applebot-Extended**](https://getkavel.com/crawlers/applebot-extended/)Apple | AI-use control | 17% (127) |
| [**GPTBot**](https://getkavel.com/crawlers/gptbot/)OpenAI | Training | 16.9% (126) |
| [**Meta-ExternalAgent**](https://getkavel.com/crawlers/meta-externalagent/)Meta | Training | 16.8% (125) |
| [**PerplexityBot**](https://getkavel.com/crawlers/perplexitybot/)Perplexity | Search | 15.3% (114) |
| [**Google-Extended**](https://getkavel.com/crawlers/google-extended/) | AI-use control | 13.5% (101) |
| [**Amazonbot**](https://getkavel.com/crawlers/amazonbot/)Amazon | General | 13.3% (99) |
| [**OAI-SearchBot**](https://getkavel.com/crawlers/oai-searchbot/)OpenAI | Search | 9.4% (70) |
| [**ChatGPT-User**](https://getkavel.com/crawlers/chatgpt-user/)OpenAI | User fetcher | 9.2% (69) |
| [**Claude-User**](https://getkavel.com/crawlers/claude-user/)Anthropic | User fetcher | 8.8% (66) |
| [**Perplexity-User**](https://getkavel.com/crawlers/perplexity-user/)Perplexity | User fetcher | 8.7% (65) |
| [**Claude-SearchBot**](https://getkavel.com/crawlers/claude-searchbot/)Anthropic | Search | 8.3% (62) |
By kind of site #
| Sites | Checked | Block any AI crawler | Block GPTBot |
|---|---|---|---|
| [US and international news](https://getkavel.com/ai-crawlers/category/us-news/) | 74 | 79.7% | 56.8% |
| [Reference](https://getkavel.com/ai-crawlers/category/reference/) | 27 | 77.8% | 48.1% |
| [UK news](https://getkavel.com/ai-crawlers/category/uk-news/) | 63 | 76.2% | 60.3% |
| [Big tech](https://getkavel.com/ai-crawlers/category/tech/) | 55 | 29.1% | 18.2% |
| [Ecommerce brands](https://getkavel.com/ai-crawlers/category/ecommerce-brands/) | 58 | 15.5% | 10.3% |
| [Travel](https://getkavel.com/ai-crawlers/category/travel/) | 29 | 13.8% | 10.3% |
| [AI companies](https://getkavel.com/ai-crawlers/category/ai/) | 44 | 13.6% | 4.5% |
| [B2B marketplaces and distributors](https://getkavel.com/ai-crawlers/category/marketplaces-b2b/) | 22 | 13.6% | 9.1% |
| [UK trade suppliers](https://getkavel.com/ai-crawlers/category/uk-trade/) | 40 | 12.5% | 5% |
| [SaaS and developer tools](https://getkavel.com/ai-crawlers/category/saas/) | 120 | 9.2% | 2.5% |
| [Education](https://getkavel.com/ai-crawlers/category/education/) | 46 | 8.7% | 4.3% |
| [Government](https://getkavel.com/ai-crawlers/category/government/) | 32 | 6.3% | 3.1% |
| [UK retail](https://getkavel.com/ai-crawlers/category/uk-retail/) | 50 | 6% | 2% |
| [Agencies](https://getkavel.com/ai-crawlers/category/agencies/) | 36 | 5.6% | 2.8% |
| [Finance](https://getkavel.com/ai-crawlers/category/finance/) | 50 | 0% | 0% |
No to training, yes to AI search #
These sites block at least one AI training crawler (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider) but leave every AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) open.
airbnb.com · alibaba.com · aliexpress.com · amplitude.com · asda.com · axios.com · bbcgoodfood.com · bostonglobe.com · businessinsider.com · calendly.com · campaignlive.co.uk · canva.com · cbsnews.com · coursera.org · descript.com · dictionary.com · ea.com · espn.com · figma.com · foodnetwork.com · fortune.com · freshbooks.com · genius.com · github.com · goodreads.com · healthline.com · indiamart.com · lakeland.co.uk · latimes.com · lonelyplanet.com · loom.com · machinemart.co.uk · marketingweek.com · mcmaster.com · medium.com · moneyweek.com · netflix.com · newstatesman.com · nintendo.com · petsathome.com · postman.com · prospectmagazine.co.uk · prweek.com · radiotimes.com · rexel.co.uk · semafor.com · skyscanner.net · slate.com · snapchat.com · speedyservices.com · techcrunch.com · techradar.com · temu.com · theregister.com · thesaurus.com · theweek.co.uk · tomshardware.com · toolstation.com · tortoisemedia.com · tripadvisor.com · trustpilot.com · vinted.com · warbyparker.com · webmd.com · which.co.uk · who.int · wpp.com · zalando.com · zoominfo.com
Lists #
## How we checked, and the limits #
- For each site we read robots.txt and apply the rules for the homepage, crawler by crawler, following the robots.txt standard (RFC 9309). A crawler is "blocked" if robots.txt asks it not to read the homepage.
- We also ask for the homepage once as GPTBot and once as ClaudeBot. Our requests come from Cloudflare's network, not from OpenAI or Anthropic, so a firewall that checks where crawlers really come from may refuse our test and still let the real crawler in.
- 199 sites are left out of the robots.txt figures: 127 turned away our request for robots.txt or sent a web page instead of the file (usually bot protection), 51 answered it with a server error, and 21 could not be reached. What their robots.txt says to real crawlers is unknown.
- 18 sites had no robots.txt rules at all (a 404 or an empty file), which under the standard means every crawler is allowed; they count as blocking nothing.
- robots.txt is a request, not a lock. This shows what sites ask, not what AI companies do.
- The list of sites was put together by Kavel; it is not a statistical sample of the web. This page was built and written with AI.
What does your site allow? #
Check your own site live, crawler by crawler. Free, no signup.