{"slug": "show-hn-which-big-sites-block-ai-training-crawlers-but-let-ai-search-in", "title": "Show HN: Which big sites block AI training crawlers but let AI search in", "summary": "A robots.txt audit of 945 well-known websites, completed 22 October 2026 and repeated monthly, found that 25.9% (193 of the 746 sites whose robots.txt could be read) block at least one of 14 AI crawlers, while 9.2% (69 sites) block AI training crawlers but leave every AI search crawler open. CCBot was the most-blocked crawler at 19.6% (146 sites), followed by Bytespider at 18.4% (137) and ClaudeBot at 18.1% (135), while GPTBot was blocked by 16.9% (126 sites). US and international news sites were the most restrictive category, with 79.7% of 74 checked blocking at least one AI crawler and 56.8% blocking GPTBot, versus 0% of 50 finance sites blocking any.", "body_md": "# Which big websites\n\nblock AI crawlers?\n\nWe checked the robots.txt rules of 945 well-known websites for 14 AI crawlers: news, retail, SaaS, finance, government and more. October 2026 check, finished 22 October 2026. We check again every month.\n\nIf a site is not one of the 945 in the study, we check it live.\n\n**25.9%**\n\nblock at least one AI crawler in robots.txt (193 of 746 sites we could read)\n\n**9.2%**\n\nblock AI training but leave every AI search crawler open (69 sites)\n\n**13.9%**\n\nturned away our test request identifying as GPTBot or ClaudeBot (see the limits below)\n\n## Blocked most often\n\n14 crawlers\n| Crawler | Kind | Sites blocking it | \n|---|---|---|\n| [**CCBot**](https://getkavel.com/crawlers/ccbot/)Common Crawl | Training | 19.6% (146) | \n| [**Bytespider**](https://getkavel.com/crawlers/bytespider/)ByteDance | Training | 18.4% (137) | \n| [**ClaudeBot**](https://getkavel.com/crawlers/claudebot/)Anthropic | Training | 18.1% (135) | \n| [**Applebot-Extended**](https://getkavel.com/crawlers/applebot-extended/)Apple | AI-use control | 17% (127) | \n| [**GPTBot**](https://getkavel.com/crawlers/gptbot/)OpenAI | Training | 16.9% (126) | \n| [**Meta-ExternalAgent**](https://getkavel.com/crawlers/meta-externalagent/)Meta | Training | 16.8% (125) | \n| [**PerplexityBot**](https://getkavel.com/crawlers/perplexitybot/)Perplexity | Search | 15.3% (114) | \n| [**Google-Extended**](https://getkavel.com/crawlers/google-extended/) | AI-use control | 13.5% (101) | \n| [**Amazonbot**](https://getkavel.com/crawlers/amazonbot/)Amazon | General | 13.3% (99) | \n| [**OAI-SearchBot**](https://getkavel.com/crawlers/oai-searchbot/)OpenAI | Search | 9.4% (70) | \n| [**ChatGPT-User**](https://getkavel.com/crawlers/chatgpt-user/)OpenAI | User fetcher | 9.2% (69) | \n| [**Claude-User**](https://getkavel.com/crawlers/claude-user/)Anthropic | User fetcher | 8.8% (66) | \n| [**Perplexity-User**](https://getkavel.com/crawlers/perplexity-user/)Perplexity | User fetcher | 8.7% (65) | \n| [**Claude-SearchBot**](https://getkavel.com/crawlers/claude-searchbot/)Anthropic | Search | 8.3% (62) | \n\n## By kind of site\n\n| Sites | Checked | Block any AI crawler | Block GPTBot | \n|---|---|---|---|\n| [US and international news](https://getkavel.com/ai-crawlers/category/us-news/) | 74 | 79.7% | 56.8% | \n| [Reference](https://getkavel.com/ai-crawlers/category/reference/) | 27 | 77.8% | 48.1% | \n| [UK news](https://getkavel.com/ai-crawlers/category/uk-news/) | 63 | 76.2% | 60.3% | \n| [Big tech](https://getkavel.com/ai-crawlers/category/tech/) | 55 | 29.1% | 18.2% | \n| [Ecommerce brands](https://getkavel.com/ai-crawlers/category/ecommerce-brands/) | 58 | 15.5% | 10.3% | \n| [Travel](https://getkavel.com/ai-crawlers/category/travel/) | 29 | 13.8% | 10.3% | \n| [AI companies](https://getkavel.com/ai-crawlers/category/ai/) | 44 | 13.6% | 4.5% | \n| [B2B marketplaces and distributors](https://getkavel.com/ai-crawlers/category/marketplaces-b2b/) | 22 | 13.6% | 9.1% | \n| [UK trade suppliers](https://getkavel.com/ai-crawlers/category/uk-trade/) | 40 | 12.5% | 5% | \n| [SaaS and developer tools](https://getkavel.com/ai-crawlers/category/saas/) | 120 | 9.2% | 2.5% | \n| [Education](https://getkavel.com/ai-crawlers/category/education/) | 46 | 8.7% | 4.3% | \n| [Government](https://getkavel.com/ai-crawlers/category/government/) | 32 | 6.3% | 3.1% | \n| [UK retail](https://getkavel.com/ai-crawlers/category/uk-retail/) | 50 | 6% | 2% | \n| [Agencies](https://getkavel.com/ai-crawlers/category/agencies/) | 36 | 5.6% | 2.8% | \n| [Finance](https://getkavel.com/ai-crawlers/category/finance/) | 50 | 0% | 0% | \n\n## No to training, yes to AI search\n\nThese sites block at least one AI training crawler (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider) but leave every AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) open.\n\n[airbnb.com](https://getkavel.com/ai-crawlers/airbnb.com/) · [alibaba.com](https://getkavel.com/ai-crawlers/alibaba.com/) · [aliexpress.com](https://getkavel.com/ai-crawlers/aliexpress.com/) · [amplitude.com](https://getkavel.com/ai-crawlers/amplitude.com/) · [asda.com](https://getkavel.com/ai-crawlers/asda.com/) · [axios.com](https://getkavel.com/ai-crawlers/axios.com/) · [bbcgoodfood.com](https://getkavel.com/ai-crawlers/bbcgoodfood.com/) · [bostonglobe.com](https://getkavel.com/ai-crawlers/bostonglobe.com/) · [businessinsider.com](https://getkavel.com/ai-crawlers/businessinsider.com/) · [calendly.com](https://getkavel.com/ai-crawlers/calendly.com/) · [campaignlive.co.uk](https://getkavel.com/ai-crawlers/campaignlive.co.uk/) · [canva.com](https://getkavel.com/ai-crawlers/canva.com/) · [cbsnews.com](https://getkavel.com/ai-crawlers/cbsnews.com/) · [coursera.org](https://getkavel.com/ai-crawlers/coursera.org/) · [descript.com](https://getkavel.com/ai-crawlers/descript.com/) · [dictionary.com](https://getkavel.com/ai-crawlers/dictionary.com/) · [ea.com](https://getkavel.com/ai-crawlers/ea.com/) · [espn.com](https://getkavel.com/ai-crawlers/espn.com/) · [figma.com](https://getkavel.com/ai-crawlers/figma.com/) · [foodnetwork.com](https://getkavel.com/ai-crawlers/foodnetwork.com/) · [fortune.com](https://getkavel.com/ai-crawlers/fortune.com/) · [freshbooks.com](https://getkavel.com/ai-crawlers/freshbooks.com/) · [genius.com](https://getkavel.com/ai-crawlers/genius.com/) · [github.com](https://getkavel.com/ai-crawlers/github.com/) · [goodreads.com](https://getkavel.com/ai-crawlers/goodreads.com/) · [healthline.com](https://getkavel.com/ai-crawlers/healthline.com/) · [indiamart.com](https://getkavel.com/ai-crawlers/indiamart.com/) · [lakeland.co.uk](https://getkavel.com/ai-crawlers/lakeland.co.uk/) · [latimes.com](https://getkavel.com/ai-crawlers/latimes.com/) · [lonelyplanet.com](https://getkavel.com/ai-crawlers/lonelyplanet.com/) · [loom.com](https://getkavel.com/ai-crawlers/loom.com/) · [machinemart.co.uk](https://getkavel.com/ai-crawlers/machinemart.co.uk/) · [marketingweek.com](https://getkavel.com/ai-crawlers/marketingweek.com/) · [mcmaster.com](https://getkavel.com/ai-crawlers/mcmaster.com/) · [medium.com](https://getkavel.com/ai-crawlers/medium.com/) · [moneyweek.com](https://getkavel.com/ai-crawlers/moneyweek.com/) · [netflix.com](https://getkavel.com/ai-crawlers/netflix.com/) · [newstatesman.com](https://getkavel.com/ai-crawlers/newstatesman.com/) · [nintendo.com](https://getkavel.com/ai-crawlers/nintendo.com/) · [petsathome.com](https://getkavel.com/ai-crawlers/petsathome.com/) · [postman.com](https://getkavel.com/ai-crawlers/postman.com/) · [prospectmagazine.co.uk](https://getkavel.com/ai-crawlers/prospectmagazine.co.uk/) · [prweek.com](https://getkavel.com/ai-crawlers/prweek.com/) · [radiotimes.com](https://getkavel.com/ai-crawlers/radiotimes.com/) · [rexel.co.uk](https://getkavel.com/ai-crawlers/rexel.co.uk/) · [semafor.com](https://getkavel.com/ai-crawlers/semafor.com/) · [skyscanner.net](https://getkavel.com/ai-crawlers/skyscanner.net/) · [slate.com](https://getkavel.com/ai-crawlers/slate.com/) · [snapchat.com](https://getkavel.com/ai-crawlers/snapchat.com/) · [speedyservices.com](https://getkavel.com/ai-crawlers/speedyservices.com/) · [techcrunch.com](https://getkavel.com/ai-crawlers/techcrunch.com/) · [techradar.com](https://getkavel.com/ai-crawlers/techradar.com/) · [temu.com](https://getkavel.com/ai-crawlers/temu.com/) · [theregister.com](https://getkavel.com/ai-crawlers/theregister.com/) · [thesaurus.com](https://getkavel.com/ai-crawlers/thesaurus.com/) · [theweek.co.uk](https://getkavel.com/ai-crawlers/theweek.co.uk/) · [tomshardware.com](https://getkavel.com/ai-crawlers/tomshardware.com/) · [toolstation.com](https://getkavel.com/ai-crawlers/toolstation.com/) · [tortoisemedia.com](https://getkavel.com/ai-crawlers/tortoisemedia.com/) · [tripadvisor.com](https://getkavel.com/ai-crawlers/tripadvisor.com/) · [trustpilot.com](https://getkavel.com/ai-crawlers/trustpilot.com/) · [vinted.com](https://getkavel.com/ai-crawlers/vinted.com/) · [warbyparker.com](https://getkavel.com/ai-crawlers/warbyparker.com/) · [webmd.com](https://getkavel.com/ai-crawlers/webmd.com/) · [which.co.uk](https://getkavel.com/ai-crawlers/which.co.uk/) · [who.int](https://getkavel.com/ai-crawlers/who.int/) · [wpp.com](https://getkavel.com/ai-crawlers/wpp.com/) · [zalando.com](https://getkavel.com/ai-crawlers/zalando.com/) · [zoominfo.com](https://getkavel.com/ai-crawlers/zoominfo.com/)\n\n## Lists\n\n## ## How we checked, and the limits\n\n- For each site we read robots.txt and apply the rules for the homepage, crawler by crawler, following the robots.txt standard (RFC 9309). A crawler is \"blocked\" if robots.txt asks it not to read the homepage.\n- We also ask for the homepage once as GPTBot and once as ClaudeBot. Our requests come from Cloudflare's network, not from OpenAI or Anthropic, so a firewall that checks where crawlers really come from may refuse our test and still let the real crawler in.\n- 199 sites are left out of the robots.txt figures: 127 turned away our request for robots.txt or sent a web page instead of the file (usually bot protection), 51 answered it with a server error, and 21 could not be reached. What their robots.txt says to real crawlers is unknown.\n- 18 sites had no robots.txt rules at all (a 404 or an empty file), which under the standard means every crawler is allowed; they count as blocking nothing.\n- robots.txt is a request, not a lock. This shows what sites ask, not what AI companies do.\n- The list of sites was put together by Kavel; it is not a statistical sample of the web. This page was built and written with AI.\n\n## What does your site allow?\n\nCheck your own site live, crawler by crawler. Free, no signup.", "url": "https://wpnews.pro/news/show-hn-which-big-sites-block-ai-training-crawlers-but-let-ai-search-in", "canonical_source": "https://getkavel.com/ai-crawlers/", "published_at": "2026-10-27 13:18:22+00:00", "updated_at": "2026-10-27 13:42:16.075669+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "artificial-intelligence"], "entities": ["CCBot", "Bytespider", "ClaudeBot", "GPTBot", "Applebot-Extended", "Meta-ExternalAgent", "PerplexityBot", "OAI-SearchBot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-which-big-sites-block-ai-training-crawlers-but-let-ai-search-in", "markdown": "https://wpnews.pro/news/show-hn-which-big-sites-block-ai-training-crawlers-but-let-ai-search-in.md", "text": "https://wpnews.pro/news/show-hn-which-big-sites-block-ai-training-crawlers-but-let-ai-search-in.txt", "jsonld": "https://wpnews.pro/news/show-hn-which-big-sites-block-ai-training-crawlers-but-let-ai-search-in.jsonld"}}