{"slug": "which-ai-crawlers-do-261-top-websites-block-study-data", "title": "Which AI crawlers do 261 top websites block? Study data", "summary": "A study of the robots.txt files of 285 of the most visited websites, 261 of which returned readable files, found that 35% block training crawlers such as GPTBot while only 21% block search-oriented agents like OAI-SearchBot. News sites accounted for 27 of the GPTBot blocks, nearly half of all GPTBot blockers in the sample, while Bytespider was blocked almost as often as CCBot despite being far less known. The raw per-site, per-crawler rule data was published openly on GitHub.", "body_md": "We checked the robots.txt files of 285 of the most visited websites to see how they treat AI crawlers. 261 of them returned a readable file. This post sums up what we found. The full report and the raw data are linked at the end.\n\nThe sample includes 48 news sites, and 27 of them block GPTBot. That is nearly half of all GPTBot blockers in the study coming from a single category. Publishers have clear reasons to keep training crawlers out, and the numbers show they act on it far more than retailers, software companies or reference sites do.\n\nThe gap between 35% and 21% matters. Many sites that shut out training bots still let search bots in, because being fetched for a live answer in ChatGPT, Perplexity or Google AI Overviews can send visitors, while training access does not. OAI-SearchBot being the least blocked agent fits that pattern. Bytespider sits at the other end: it is blocked almost as often as CCBot even though it is far less known, likely because it has a reputation for ignoring rules.\n\nThe CSV with every site, every crawler and every rule is open on GitHub: [ai-crawler-access-study](https://github.com/airanked/ai-crawler-access-study)\n\nThe full write up, including method and per category breakdowns, is here: [AI crawler access report](https://airanked.ai/research/ai-crawler-access-report)\n\nIf you want to check your own robots.txt against these crawlers, AI Ranked has a [free in-browser robots.txt checker](https://airanked.ai/tools/robots-txt-ai-checker).", "url": "https://wpnews.pro/news/which-ai-crawlers-do-261-top-websites-block-study-data", "canonical_source": "https://dev.to/airanked/which-ai-crawlers-do-261-top-websites-block-study-data-41e3", "published_at": "2026-10-15 13:35:54+00:00", "updated_at": "2026-10-15 13:49:28.359406+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "structured-data"], "entities": ["GPTBot", "OAI-SearchBot", "Bytespider", "CCBot", "ChatGPT", "Perplexity", "Google AI Overviews", "AI Ranked"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/which-ai-crawlers-do-261-top-websites-block-study-data", "markdown": "https://wpnews.pro/news/which-ai-crawlers-do-261-top-websites-block-study-data.md", "text": "https://wpnews.pro/news/which-ai-crawlers-do-261-top-websites-block-study-data.txt", "jsonld": "https://wpnews.pro/news/which-ai-crawlers-do-261-top-websites-block-study-data.jsonld"}}