Show HN: Which big sites block AI training crawlers but let AI search in

A robots.txt audit of 945 well-known websites, completed 22 October 2026 and repeated monthly, found that 25.9% (193 of the 746 sites whose robots.txt could be read) block at least one of 14 AI crawlers, while 9.2% (69 sites) block AI training crawlers but leave every AI search crawler open. CCBot was the most-blocked crawler at 19.6% (146 sites), followed by Bytespider at 18.4% (137) and ClaudeBot at 18.1% (135), while GPTBot was blocked by 16.9% (126 sites). US and international news sites were the most restrictive category, with 79.7% of 74 checked blocking at least one AI crawler and 56.8% blocking GPTBot, versus 0% of 50 finance sites blocking any.

Which big websites block AI crawlers? We checked the robots.txt rules of 945 well-known websites for 14 AI crawlers: news, retail, SaaS, finance, government and more. October 2026 check, finished 22 October 2026. We check again every month. If a site is not one of the 945 in the study, we check it live. 25.9% block at least one AI crawler in robots.txt 193 of 746 sites we could read 9.2% block AI training but leave every AI search crawler open 69 sites 13.9% turned away our test request identifying as GPTBot or ClaudeBot see the limits below Blocked most often 14 crawlers | Crawler | Kind | Sites blocking it | |---|---|---| | CCBot https://getkavel.com/crawlers/ccbot/ Common Crawl | Training | 19.6% 146 | | Bytespider https://getkavel.com/crawlers/bytespider/ ByteDance | Training | 18.4% 137 | | ClaudeBot https://getkavel.com/crawlers/claudebot/ Anthropic | Training | 18.1% 135 | | Applebot-Extended https://getkavel.com/crawlers/applebot-extended/ Apple | AI-use control | 17% 127 | | GPTBot https://getkavel.com/crawlers/gptbot/ OpenAI | Training | 16.9% 126 | | Meta-ExternalAgent https://getkavel.com/crawlers/meta-externalagent/ Meta | Training | 16.8% 125 | | PerplexityBot https://getkavel.com/crawlers/perplexitybot/ Perplexity | Search | 15.3% 114 | | Google-Extended https://getkavel.com/crawlers/google-extended/ | AI-use control | 13.5% 101 | | Amazonbot https://getkavel.com/crawlers/amazonbot/ Amazon | General | 13.3% 99 | | OAI-SearchBot https://getkavel.com/crawlers/oai-searchbot/ OpenAI | Search | 9.4% 70 | | ChatGPT-User https://getkavel.com/crawlers/chatgpt-user/ OpenAI | User fetcher | 9.2% 69 | | Claude-User https://getkavel.com/crawlers/claude-user/ Anthropic | User fetcher | 8.8% 66 | | Perplexity-User https://getkavel.com/crawlers/perplexity-user/ Perplexity | User fetcher | 8.7% 65 | | Claude-SearchBot https://getkavel.com/crawlers/claude-searchbot/ Anthropic | Search | 8.3% 62 | By kind of site | Sites | Checked | Block any AI crawler | Block GPTBot | |---|---|---|---| | US and international news https://getkavel.com/ai-crawlers/category/us-news/ | 74 | 79.7% | 56.8% | | Reference https://getkavel.com/ai-crawlers/category/reference/ | 27 | 77.8% | 48.1% | | UK news https://getkavel.com/ai-crawlers/category/uk-news/ | 63 | 76.2% | 60.3% | | Big tech https://getkavel.com/ai-crawlers/category/tech/ | 55 | 29.1% | 18.2% | | Ecommerce brands https://getkavel.com/ai-crawlers/category/ecommerce-brands/ | 58 | 15.5% | 10.3% | | Travel https://getkavel.com/ai-crawlers/category/travel/ | 29 | 13.8% | 10.3% | | AI companies https://getkavel.com/ai-crawlers/category/ai/ | 44 | 13.6% | 4.5% | | B2B marketplaces and distributors https://getkavel.com/ai-crawlers/category/marketplaces-b2b/ | 22 | 13.6% | 9.1% | | UK trade suppliers https://getkavel.com/ai-crawlers/category/uk-trade/ | 40 | 12.5% | 5% | | SaaS and developer tools https://getkavel.com/ai-crawlers/category/saas/ | 120 | 9.2% | 2.5% | | Education https://getkavel.com/ai-crawlers/category/education/ | 46 | 8.7% | 4.3% | | Government https://getkavel.com/ai-crawlers/category/government/ | 32 | 6.3% | 3.1% | | UK retail https://getkavel.com/ai-crawlers/category/uk-retail/ | 50 | 6% | 2% | | Agencies https://getkavel.com/ai-crawlers/category/agencies/ | 36 | 5.6% | 2.8% | | Finance https://getkavel.com/ai-crawlers/category/finance/ | 50 | 0% | 0% | No to training, yes to AI search These sites block at least one AI training crawler GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider but leave every AI search crawler OAI-SearchBot, Claude-SearchBot, PerplexityBot open. airbnb.com https://getkavel.com/ai-crawlers/airbnb.com/ · alibaba.com https://getkavel.com/ai-crawlers/alibaba.com/ · aliexpress.com https://getkavel.com/ai-crawlers/aliexpress.com/ · amplitude.com https://getkavel.com/ai-crawlers/amplitude.com/ · asda.com https://getkavel.com/ai-crawlers/asda.com/ · axios.com https://getkavel.com/ai-crawlers/axios.com/ · bbcgoodfood.com https://getkavel.com/ai-crawlers/bbcgoodfood.com/ · bostonglobe.com https://getkavel.com/ai-crawlers/bostonglobe.com/ · businessinsider.com https://getkavel.com/ai-crawlers/businessinsider.com/ · calendly.com https://getkavel.com/ai-crawlers/calendly.com/ · campaignlive.co.uk https://getkavel.com/ai-crawlers/campaignlive.co.uk/ · canva.com https://getkavel.com/ai-crawlers/canva.com/ · cbsnews.com https://getkavel.com/ai-crawlers/cbsnews.com/ · coursera.org https://getkavel.com/ai-crawlers/coursera.org/ · descript.com https://getkavel.com/ai-crawlers/descript.com/ · dictionary.com https://getkavel.com/ai-crawlers/dictionary.com/ · ea.com https://getkavel.com/ai-crawlers/ea.com/ · espn.com https://getkavel.com/ai-crawlers/espn.com/ · figma.com https://getkavel.com/ai-crawlers/figma.com/ · foodnetwork.com https://getkavel.com/ai-crawlers/foodnetwork.com/ · fortune.com https://getkavel.com/ai-crawlers/fortune.com/ · freshbooks.com https://getkavel.com/ai-crawlers/freshbooks.com/ · genius.com https://getkavel.com/ai-crawlers/genius.com/ · github.com https://getkavel.com/ai-crawlers/github.com/ · goodreads.com https://getkavel.com/ai-crawlers/goodreads.com/ · healthline.com https://getkavel.com/ai-crawlers/healthline.com/ · indiamart.com https://getkavel.com/ai-crawlers/indiamart.com/ · lakeland.co.uk https://getkavel.com/ai-crawlers/lakeland.co.uk/ · latimes.com https://getkavel.com/ai-crawlers/latimes.com/ · lonelyplanet.com https://getkavel.com/ai-crawlers/lonelyplanet.com/ · loom.com https://getkavel.com/ai-crawlers/loom.com/ · machinemart.co.uk https://getkavel.com/ai-crawlers/machinemart.co.uk/ · marketingweek.com https://getkavel.com/ai-crawlers/marketingweek.com/ · mcmaster.com https://getkavel.com/ai-crawlers/mcmaster.com/ · medium.com https://getkavel.com/ai-crawlers/medium.com/ · moneyweek.com https://getkavel.com/ai-crawlers/moneyweek.com/ · netflix.com https://getkavel.com/ai-crawlers/netflix.com/ · newstatesman.com https://getkavel.com/ai-crawlers/newstatesman.com/ · nintendo.com https://getkavel.com/ai-crawlers/nintendo.com/ · petsathome.com https://getkavel.com/ai-crawlers/petsathome.com/ · postman.com https://getkavel.com/ai-crawlers/postman.com/ · prospectmagazine.co.uk https://getkavel.com/ai-crawlers/prospectmagazine.co.uk/ · prweek.com https://getkavel.com/ai-crawlers/prweek.com/ · radiotimes.com https://getkavel.com/ai-crawlers/radiotimes.com/ · rexel.co.uk https://getkavel.com/ai-crawlers/rexel.co.uk/ · semafor.com https://getkavel.com/ai-crawlers/semafor.com/ · skyscanner.net https://getkavel.com/ai-crawlers/skyscanner.net/ · slate.com https://getkavel.com/ai-crawlers/slate.com/ · snapchat.com https://getkavel.com/ai-crawlers/snapchat.com/ · speedyservices.com https://getkavel.com/ai-crawlers/speedyservices.com/ · techcrunch.com https://getkavel.com/ai-crawlers/techcrunch.com/ · techradar.com https://getkavel.com/ai-crawlers/techradar.com/ · temu.com https://getkavel.com/ai-crawlers/temu.com/ · theregister.com https://getkavel.com/ai-crawlers/theregister.com/ · thesaurus.com https://getkavel.com/ai-crawlers/thesaurus.com/ · theweek.co.uk https://getkavel.com/ai-crawlers/theweek.co.uk/ · tomshardware.com https://getkavel.com/ai-crawlers/tomshardware.com/ · toolstation.com https://getkavel.com/ai-crawlers/toolstation.com/ · tortoisemedia.com https://getkavel.com/ai-crawlers/tortoisemedia.com/ · tripadvisor.com https://getkavel.com/ai-crawlers/tripadvisor.com/ · trustpilot.com https://getkavel.com/ai-crawlers/trustpilot.com/ · vinted.com https://getkavel.com/ai-crawlers/vinted.com/ · warbyparker.com https://getkavel.com/ai-crawlers/warbyparker.com/ · webmd.com https://getkavel.com/ai-crawlers/webmd.com/ · which.co.uk https://getkavel.com/ai-crawlers/which.co.uk/ · who.int https://getkavel.com/ai-crawlers/who.int/ · wpp.com https://getkavel.com/ai-crawlers/wpp.com/ · zalando.com https://getkavel.com/ai-crawlers/zalando.com/ · zoominfo.com https://getkavel.com/ai-crawlers/zoominfo.com/ Lists How we checked, and the limits - For each site we read robots.txt and apply the rules for the homepage, crawler by crawler, following the robots.txt standard RFC 9309 . A crawler is "blocked" if robots.txt asks it not to read the homepage. - We also ask for the homepage once as GPTBot and once as ClaudeBot. Our requests come from Cloudflare's network, not from OpenAI or Anthropic, so a firewall that checks where crawlers really come from may refuse our test and still let the real crawler in. - 199 sites are left out of the robots.txt figures: 127 turned away our request for robots.txt or sent a web page instead of the file usually bot protection , 51 answered it with a server error, and 21 could not be reached. What their robots.txt says to real crawlers is unknown. - 18 sites had no robots.txt rules at all a 404 or an empty file , which under the standard means every crawler is allowed; they count as blocking nothing. - robots.txt is a request, not a lock. This shows what sites ask, not what AI companies do. - The list of sites was put together by Kavel; it is not a statistical sample of the web. This page was built and written with AI. What does your site allow? Check your own site live, crawler by crawler. Free, no signup.