AI companies now run their own crawlers, and most identify themselves with a user-agent name you can put in robots.txt. This guide explains what the main ones do, how training crawlers differ from search and answer crawlers, and gives copy-paste rules for three common policies. Confirm details against each vendor's current documentation before you deploy.
First, a limit: robots.txt is a request, not access control. Well-behaved crawlers honor it. Scrapers that ignore it are not stopped by it, and a blocked URL can still appear in search results if other sites link to it.
GPTBot is OpenAI's crawler. OpenAI documents it as collecting web content that may be used to train its models.
OAI-SearchBot is OpenAI's crawler for ChatGPT search. It indexes pages so ChatGPT can show them in search results and cite them. It is separate from GPTBot, so you can allow one and block the other.
ChatGPT-User runs when a person asks ChatGPT to open a page during a conversation. It is user-triggered rather than a bulk crawl, and OpenAI notes that robots.txt rules may not apply to these requests.
ClaudeBot is Anthropic's crawler, used to collect content that may be used for training. Anthropic also documents separate agents for search and user-requested fetching, so check its current list before writing rules for them.
PerplexityBot indexes pages for Perplexity's answers and citations. Perplexity says it respects robots.txt. Its user-triggered fetcher, Perplexity-User, behaves like ChatGPT-User.
Google-Extended is not a crawler. It is a robots.txt token that controls whether content Google has already crawled can be used to train Gemini and for certain grounding features. Googlebot still crawls your site for Search whether or not you block this token, so blocking Google-Extended does not remove you from Google Search.
The useful split is purpose.
If you want citations and referral traffic from AI answers, allow the search crawlers. If your objection is only to training, block the training agents and keep the search agents open.
Allow everything. This is the default, written out explicitly:
User-agent: *
Allow: /
Block training, allow search and answers:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Block all of these AI crawlers:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
A crawler obeys the group that names it and ignores the * group. So the bots you block stay blocked even though * allows everything. The * group applies to crawlers you did not name, such as Googlebot and Bingbot.
User-agent: * and Disallow: /. https://example.com/robots.txt. Each subdomain, such as blog.example.com, needs its own file./Blog/ and /blog/ are different paths. Bot names are case-insensitive, but copy the vendor's spelling anyway.
from urllib.robotparser import RobotFileParser
rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
for bot in ["GPTBot", "OAI-SearchBot", "PerplexityBot"]:
print(bot, rp.can_fetch(bot, "https://example.com/blog/post"))
Its wildcard handling differs from Google's, so verify key paths with a second parser.
Decide your policy first: allow all, block training only, or block everything. Write named groups for the bots you care about, host the file at the root, and verify it with logs and a parser.
If you would rather generate these groups than type them, the free generator at https://citescore.vercel.app/robots-txt-ai-crawler-generator builds them.