# Show HN: Which big sites block AI training crawlers but let AI search in > Source: > Published: 2026-10-27 13:18:22+00:00 # Which big websites block AI crawlers? We checked the robots.txt rules of 945 well-known websites for 14 AI crawlers: news, retail, SaaS, finance, government and more. October 2026 check, finished 22 October 2026. We check again every month. If a site is not one of the 945 in the study, we check it live. **25.9%** block at least one AI crawler in robots.txt (193 of 746 sites we could read) **9.2%** block AI training but leave every AI search crawler open (69 sites) **13.9%** turned away our test request identifying as GPTBot or ClaudeBot (see the limits below) ## Blocked most often 14 crawlers | Crawler | Kind | Sites blocking it | |---|---|---| | [**CCBot**](https://getkavel.com/crawlers/ccbot/)Common Crawl | Training | 19.6% (146) | | [**Bytespider**](https://getkavel.com/crawlers/bytespider/)ByteDance | Training | 18.4% (137) | | [**ClaudeBot**](https://getkavel.com/crawlers/claudebot/)Anthropic | Training | 18.1% (135) | | [**Applebot-Extended**](https://getkavel.com/crawlers/applebot-extended/)Apple | AI-use control | 17% (127) | | [**GPTBot**](https://getkavel.com/crawlers/gptbot/)OpenAI | Training | 16.9% (126) | | [**Meta-ExternalAgent**](https://getkavel.com/crawlers/meta-externalagent/)Meta | Training | 16.8% (125) | | [**PerplexityBot**](https://getkavel.com/crawlers/perplexitybot/)Perplexity | Search | 15.3% (114) | | [**Google-Extended**](https://getkavel.com/crawlers/google-extended/) | AI-use control | 13.5% (101) | | [**Amazonbot**](https://getkavel.com/crawlers/amazonbot/)Amazon | General | 13.3% (99) | | [**OAI-SearchBot**](https://getkavel.com/crawlers/oai-searchbot/)OpenAI | Search | 9.4% (70) | | [**ChatGPT-User**](https://getkavel.com/crawlers/chatgpt-user/)OpenAI | User fetcher | 9.2% (69) | | [**Claude-User**](https://getkavel.com/crawlers/claude-user/)Anthropic | User fetcher | 8.8% (66) | | [**Perplexity-User**](https://getkavel.com/crawlers/perplexity-user/)Perplexity | User fetcher | 8.7% (65) | | [**Claude-SearchBot**](https://getkavel.com/crawlers/claude-searchbot/)Anthropic | Search | 8.3% (62) | ## By kind of site | Sites | Checked | Block any AI crawler | Block GPTBot | |---|---|---|---| | [US and international news](https://getkavel.com/ai-crawlers/category/us-news/) | 74 | 79.7% | 56.8% | | [Reference](https://getkavel.com/ai-crawlers/category/reference/) | 27 | 77.8% | 48.1% | | [UK news](https://getkavel.com/ai-crawlers/category/uk-news/) | 63 | 76.2% | 60.3% | | [Big tech](https://getkavel.com/ai-crawlers/category/tech/) | 55 | 29.1% | 18.2% | | [Ecommerce brands](https://getkavel.com/ai-crawlers/category/ecommerce-brands/) | 58 | 15.5% | 10.3% | | [Travel](https://getkavel.com/ai-crawlers/category/travel/) | 29 | 13.8% | 10.3% | | [AI companies](https://getkavel.com/ai-crawlers/category/ai/) | 44 | 13.6% | 4.5% | | [B2B marketplaces and distributors](https://getkavel.com/ai-crawlers/category/marketplaces-b2b/) | 22 | 13.6% | 9.1% | | [UK trade suppliers](https://getkavel.com/ai-crawlers/category/uk-trade/) | 40 | 12.5% | 5% | | [SaaS and developer tools](https://getkavel.com/ai-crawlers/category/saas/) | 120 | 9.2% | 2.5% | | [Education](https://getkavel.com/ai-crawlers/category/education/) | 46 | 8.7% | 4.3% | | [Government](https://getkavel.com/ai-crawlers/category/government/) | 32 | 6.3% | 3.1% | | [UK retail](https://getkavel.com/ai-crawlers/category/uk-retail/) | 50 | 6% | 2% | | [Agencies](https://getkavel.com/ai-crawlers/category/agencies/) | 36 | 5.6% | 2.8% | | [Finance](https://getkavel.com/ai-crawlers/category/finance/) | 50 | 0% | 0% | ## No to training, yes to AI search These sites block at least one AI training crawler (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider) but leave every AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) open. [airbnb.com](https://getkavel.com/ai-crawlers/airbnb.com/) · [alibaba.com](https://getkavel.com/ai-crawlers/alibaba.com/) · [aliexpress.com](https://getkavel.com/ai-crawlers/aliexpress.com/) · [amplitude.com](https://getkavel.com/ai-crawlers/amplitude.com/) · [asda.com](https://getkavel.com/ai-crawlers/asda.com/) · [axios.com](https://getkavel.com/ai-crawlers/axios.com/) · [bbcgoodfood.com](https://getkavel.com/ai-crawlers/bbcgoodfood.com/) · [bostonglobe.com](https://getkavel.com/ai-crawlers/bostonglobe.com/) · [businessinsider.com](https://getkavel.com/ai-crawlers/businessinsider.com/) · [calendly.com](https://getkavel.com/ai-crawlers/calendly.com/) · [campaignlive.co.uk](https://getkavel.com/ai-crawlers/campaignlive.co.uk/) · [canva.com](https://getkavel.com/ai-crawlers/canva.com/) · [cbsnews.com](https://getkavel.com/ai-crawlers/cbsnews.com/) · [coursera.org](https://getkavel.com/ai-crawlers/coursera.org/) · [descript.com](https://getkavel.com/ai-crawlers/descript.com/) · [dictionary.com](https://getkavel.com/ai-crawlers/dictionary.com/) · [ea.com](https://getkavel.com/ai-crawlers/ea.com/) · [espn.com](https://getkavel.com/ai-crawlers/espn.com/) · [figma.com](https://getkavel.com/ai-crawlers/figma.com/) · [foodnetwork.com](https://getkavel.com/ai-crawlers/foodnetwork.com/) · [fortune.com](https://getkavel.com/ai-crawlers/fortune.com/) · [freshbooks.com](https://getkavel.com/ai-crawlers/freshbooks.com/) · [genius.com](https://getkavel.com/ai-crawlers/genius.com/) · [github.com](https://getkavel.com/ai-crawlers/github.com/) · [goodreads.com](https://getkavel.com/ai-crawlers/goodreads.com/) · [healthline.com](https://getkavel.com/ai-crawlers/healthline.com/) · [indiamart.com](https://getkavel.com/ai-crawlers/indiamart.com/) · [lakeland.co.uk](https://getkavel.com/ai-crawlers/lakeland.co.uk/) · [latimes.com](https://getkavel.com/ai-crawlers/latimes.com/) · [lonelyplanet.com](https://getkavel.com/ai-crawlers/lonelyplanet.com/) · [loom.com](https://getkavel.com/ai-crawlers/loom.com/) · [machinemart.co.uk](https://getkavel.com/ai-crawlers/machinemart.co.uk/) · [marketingweek.com](https://getkavel.com/ai-crawlers/marketingweek.com/) · [mcmaster.com](https://getkavel.com/ai-crawlers/mcmaster.com/) · [medium.com](https://getkavel.com/ai-crawlers/medium.com/) · [moneyweek.com](https://getkavel.com/ai-crawlers/moneyweek.com/) · [netflix.com](https://getkavel.com/ai-crawlers/netflix.com/) · [newstatesman.com](https://getkavel.com/ai-crawlers/newstatesman.com/) · [nintendo.com](https://getkavel.com/ai-crawlers/nintendo.com/) · [petsathome.com](https://getkavel.com/ai-crawlers/petsathome.com/) · [postman.com](https://getkavel.com/ai-crawlers/postman.com/) · [prospectmagazine.co.uk](https://getkavel.com/ai-crawlers/prospectmagazine.co.uk/) · [prweek.com](https://getkavel.com/ai-crawlers/prweek.com/) · [radiotimes.com](https://getkavel.com/ai-crawlers/radiotimes.com/) · [rexel.co.uk](https://getkavel.com/ai-crawlers/rexel.co.uk/) · [semafor.com](https://getkavel.com/ai-crawlers/semafor.com/) · [skyscanner.net](https://getkavel.com/ai-crawlers/skyscanner.net/) · [slate.com](https://getkavel.com/ai-crawlers/slate.com/) · [snapchat.com](https://getkavel.com/ai-crawlers/snapchat.com/) · [speedyservices.com](https://getkavel.com/ai-crawlers/speedyservices.com/) · [techcrunch.com](https://getkavel.com/ai-crawlers/techcrunch.com/) · [techradar.com](https://getkavel.com/ai-crawlers/techradar.com/) · [temu.com](https://getkavel.com/ai-crawlers/temu.com/) · [theregister.com](https://getkavel.com/ai-crawlers/theregister.com/) · [thesaurus.com](https://getkavel.com/ai-crawlers/thesaurus.com/) · [theweek.co.uk](https://getkavel.com/ai-crawlers/theweek.co.uk/) · [tomshardware.com](https://getkavel.com/ai-crawlers/tomshardware.com/) · [toolstation.com](https://getkavel.com/ai-crawlers/toolstation.com/) · [tortoisemedia.com](https://getkavel.com/ai-crawlers/tortoisemedia.com/) · [tripadvisor.com](https://getkavel.com/ai-crawlers/tripadvisor.com/) · [trustpilot.com](https://getkavel.com/ai-crawlers/trustpilot.com/) · [vinted.com](https://getkavel.com/ai-crawlers/vinted.com/) · [warbyparker.com](https://getkavel.com/ai-crawlers/warbyparker.com/) · [webmd.com](https://getkavel.com/ai-crawlers/webmd.com/) · [which.co.uk](https://getkavel.com/ai-crawlers/which.co.uk/) · [who.int](https://getkavel.com/ai-crawlers/who.int/) · [wpp.com](https://getkavel.com/ai-crawlers/wpp.com/) · [zalando.com](https://getkavel.com/ai-crawlers/zalando.com/) · [zoominfo.com](https://getkavel.com/ai-crawlers/zoominfo.com/) ## Lists ## ## How we checked, and the limits - For each site we read robots.txt and apply the rules for the homepage, crawler by crawler, following the robots.txt standard (RFC 9309). A crawler is "blocked" if robots.txt asks it not to read the homepage. - We also ask for the homepage once as GPTBot and once as ClaudeBot. Our requests come from Cloudflare's network, not from OpenAI or Anthropic, so a firewall that checks where crawlers really come from may refuse our test and still let the real crawler in. - 199 sites are left out of the robots.txt figures: 127 turned away our request for robots.txt or sent a web page instead of the file (usually bot protection), 51 answered it with a server error, and 21 could not be reached. What their robots.txt says to real crawlers is unknown. - 18 sites had no robots.txt rules at all (a 404 or an empty file), which under the standard means every crawler is allowed; they count as blocking nothing. - robots.txt is a request, not a lock. This shows what sites ask, not what AI companies do. - The list of sites was put together by Kavel; it is not a statistical sample of the web. This page was built and written with AI. ## What does your site allow? Check your own site live, crawler by crawler. Free, no signup.