robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot explained

A developer published a guide explaining how to use robots.txt to control AI crawlers, distinguishing training crawlers like GPTBot, ClaudeBot and Google-Extended from search and answer crawlers such as OAI-SearchBot and PerplexityBot. The guide provides copy-paste rules for three policies — allow everything, block training while allowing search, and block all listed AI crawlers — and notes that robots.txt is a request rather than access control, since scrapers that ignore it are not stopped. It also includes a Python snippet using urllib.robotparser to verify rules and warns that its wildcard handling differs from Google's.

AI companies now run their own crawlers, and most identify themselves with a user-agent name you can put in robots.txt. This guide explains what the main ones do, how training crawlers differ from search and answer crawlers, and gives copy-paste rules for three common policies. Confirm details against each vendor's current documentation before you deploy. First, a limit: robots.txt is a request, not access control. Well-behaved crawlers honor it. Scrapers that ignore it are not stopped by it, and a blocked URL can still appear in search results if other sites link to it. GPTBot is OpenAI's crawler. OpenAI documents it as collecting web content that may be used to train its models. OAI-SearchBot is OpenAI's crawler for ChatGPT search. It indexes pages so ChatGPT can show them in search results and cite them. It is separate from GPTBot, so you can allow one and block the other. ChatGPT-User runs when a person asks ChatGPT to open a page during a conversation. It is user-triggered rather than a bulk crawl, and OpenAI notes that robots.txt rules may not apply to these requests. ClaudeBot is Anthropic's crawler, used to collect content that may be used for training. Anthropic also documents separate agents for search and user-requested fetching, so check its current list before writing rules for them. PerplexityBot indexes pages for Perplexity's answers and citations. Perplexity says it respects robots.txt. Its user-triggered fetcher, Perplexity-User, behaves like ChatGPT-User. Google-Extended is not a crawler. It is a robots.txt token that controls whether content Google has already crawled can be used to train Gemini and for certain grounding features. Googlebot still crawls your site for Search whether or not you block this token, so blocking Google-Extended does not remove you from Google Search. The useful split is purpose. If you want citations and referral traffic from AI answers, allow the search crawlers. If your objection is only to training, block the training agents and keep the search agents open. Allow everything. This is the default, written out explicitly: User-agent: Allow: / Block training, allow search and answers: User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot User-agent: PerplexityBot Allow: / User-agent: Allow: / Block all of these AI crawlers: User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Google-Extended Disallow: / User-agent: Allow: / A crawler obeys the group that names it and ignores the group. So the bots you block stay blocked even though allows everything. The group applies to crawlers you did not name, such as Googlebot and Bingbot. User-agent: and Disallow: / . https://example.com/robots.txt . Each subdomain, such as blog.example.com , needs its own file. /Blog/ and /blog/ are different paths. Bot names are case-insensitive, but copy the vendor's spelling anyway. python from urllib.robotparser import RobotFileParser rp = RobotFileParser "https://example.com/robots.txt" rp.read for bot in "GPTBot", "OAI-SearchBot", "PerplexityBot" : print bot, rp.can fetch bot, "https://example.com/blog/post" Its wildcard handling differs from Google's, so verify key paths with a second parser. Decide your policy first: allow all, block training only, or block everything. Write named groups for the bots you care about, host the file at the root, and verify it with logs and a parser. If you would rather generate these groups than type them, the free generator at https://citescore.vercel.app/robots-txt-ai-crawler-generator https://citescore.vercel.app/robots-txt-ai-crawler-generator builds them.