We checked the robots.txt files of 285 of the most visited websites to see how they treat AI crawlers. 261 of them returned a readable file. This post sums up what we found. The full report and the raw data are linked at the end.
The sample includes 48 news sites, and 27 of them block GPTBot. That is nearly half of all GPTBot blockers in the study coming from a single category. Publishers have clear reasons to keep training crawlers out, and the numbers show they act on it far more than retailers, software companies or reference sites do.
The gap between 35% and 21% matters. Many sites that shut out training bots still let search bots in, because being fetched for a live answer in ChatGPT, Perplexity or Google AI Overviews can send visitors, while training access does not. OAI-SearchBot being the least blocked agent fits that pattern. Bytespider sits at the other end: it is blocked almost as often as CCBot even though it is far less known, likely because it has a reputation for ignoring rules.
The CSV with every site, every crawler and every rule is open on GitHub: [ai-crawler-access-study](https://github.com/airanked/ai-crawler-access-study)
The full write up, including method and per category breakdowns, is here: [AI crawler access report](https://airanked.ai/research/ai-crawler-access-report)
If you want to check your own robots.txt against these crawlers, AI Ranked has a [free in-browser robots.txt checker](https://airanked.ai/tools/robots-txt-ai-checker).