Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it

A developer argues that blocking AI crawlers in robots.txt should be a per-purpose decision rather than a blanket per-company block, noting that operators now publish distinct user agents for training (GPTBot, ClaudeBot, CCBot), search (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and user-triggered fetches (ChatGPT-User, Claude-User), plus opt-outs like Google-Extended and Applebot-Extended. The piece recommends disallowing training crawlers individually while leaving the wildcard allowed so search and user-triggered agents still reach the site, and offers a short Python RobotFileParser script to verify which agents a live robots.txt actually permits.

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com https://seotto.com A couple of years ago, robots.txt was mostly about search engines. Now a typical site's logs include a whole list of AI-related crawlers: OpenAI's, Anthropic's, Common Crawl's, Perplexity's, Apple's, ByteDance's and others. Many sites added blanket blocks in a hurry. Others never touched their robots.txt . Should you block them? I think this is a genuinely open question, and the answer has changed as the bots themselves have changed. Let's argue about it. The big operators now publish several user agents with different purposes. Roughly: GPTBot , Anthropic's ClaudeBot , Common Crawl's CCBot used by many model builders . OAI-SearchBot , PerplexityBot , Claude-SearchBot . ChatGPT-User , Claude-User . Google-Extended and Applebot-Extended are names you can disallow in robots.txt to opt out of your content being used for those companies' AI models, without affecting Google Search or Apple's search features. Names and purposes change, so check each operator's current documentation before relying on a list. The point is that "block AI" now means several separate decisions. Consent. Your content was written for readers, and nobody asked whether it could be used to train a commercial model. Blocking training crawlers is the clearest signal you can give. No traffic in return. Training a model on your article doesn't send anyone to your site. If the model can then answer the question your article answers, it may reduce your traffic. Server cost. Some AI crawlers are aggressive, request huge numbers of pages, and ignore crawl-delay hints. Smaller sites have seen real bandwidth and CPU costs from them. Leverage. Publishers negotiating licensing deals have argued that blocking is the only bargaining position content owners have. AI answers are a discovery channel now. People ask assistants for recommendations, documentation and comparisons. Products that cite sources can only cite pages their search crawlers can read. Block the search agents and you're invisible there. Developer content wants to be found. For documentation, open source projects and developer tools, being the source an assistant learned from, or cites, is often the whole point of publishing. Blocking doesn't remove what's already collected. Pages crawled before you added the rule may already be in datasets. A new block only affects future crawling. robots.txt is voluntary. Well-behaved crawlers follow it. Badly behaved ones don't, and some scrapers simply pretend to be a browser. A block mostly affects the operators who were already playing fair. Before you tighten robots.txt, it helps to see what is already blocked. seotto.com https://seotto.com checks for indexing blocks and robots regressions, including rules you didn't mean to ship. For most sites, I'd make the decision per purpose, not per company: User-agent: disallow added in a hurry to stop AI bots also removes you from regular search. An example of the split approach: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Allow: / Search and user-triggered agents fall under and stay allowed. Before changing anything, fetch your live file and test a few agents against it. Python's standard library can do it: python from urllib.robotparser import RobotFileParser rp = RobotFileParser "https://example.com/robots.txt" ; rp.read for ua in "Googlebot", "Bingbot", "GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "CCBot" : print f"{ua:15s}", rp.can fetch ua, "https://example.com/blog/some-post" It's a quick way to catch the worst outcome, which is blocking Googlebot while trying to block something else. Whatever you decide, put a reminder in the calendar. The list of crawlers, what each one is used for, and the products that cite sources are all changing quickly. A rule written for last year's bots may block a search agent that didn't exist then, or miss a new training crawler entirely. A quarterly look at your logs and at each operator's current documentation keeps the file honest. Before deciding, look at what these bots actually do on your site. How much of your crawl traffic they make up, and which pages they hit, is a ten-minute log query. Measuring that kind of thing from real logs, rather than guessing, is what we prefer at seotto.com https://seotto.com . So what's in your robots.txt today: block everything AI, allow everything, or a split like this? And has anyone measured traffic from AI answers well enough to know whether allowing them paid off?