# robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot explained

> Source: <https://dev.to/citescore/robotstxt-for-ai-crawlers-gptbot-claudebot-perplexitybot-explained-46mc>
> Published: 2026-10-10 20:43:44+00:00

AI companies now run their own crawlers, and most identify themselves with a user-agent name you can put in robots.txt. This guide explains what the main ones do, how training crawlers differ from search and answer crawlers, and gives copy-paste rules for three common policies. Confirm details against each vendor's current documentation before you deploy.

First, a limit: robots.txt is a request, not access control. Well-behaved crawlers honor it. Scrapers that ignore it are not stopped by it, and a blocked URL can still appear in search results if other sites link to it.

**GPTBot** is OpenAI's crawler. OpenAI documents it as collecting web content that may be used to train its models.

**OAI-SearchBot** is OpenAI's crawler for ChatGPT search. It indexes pages so ChatGPT can show them in search results and cite them. It is separate from GPTBot, so you can allow one and block the other.

**ChatGPT-User** runs when a person asks ChatGPT to open a page during a conversation. It is user-triggered rather than a bulk crawl, and OpenAI notes that robots.txt rules may not apply to these requests.

**ClaudeBot** is Anthropic's crawler, used to collect content that may be used for training. Anthropic also documents separate agents for search and user-requested fetching, so check its current list before writing rules for them.

**PerplexityBot** indexes pages for Perplexity's answers and citations. Perplexity says it respects robots.txt. Its user-triggered fetcher, Perplexity-User, behaves like ChatGPT-User.

**Google-Extended** is not a crawler. It is a robots.txt token that controls whether content Google has already crawled can be used to train Gemini and for certain grounding features. Googlebot still crawls your site for Search whether or not you block this token, so blocking Google-Extended does not remove you from Google Search.

The useful split is purpose.

If you want citations and referral traffic from AI answers, allow the search crawlers. If your objection is only to training, block the training agents and keep the search agents open.

**Allow everything.** This is the default, written out explicitly:

```
User-agent: *
Allow: /
```

**Block training, allow search and answers:**

```
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /
```

**Block all of these AI crawlers:**

```
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
```

A crawler obeys the group that names it and ignores the `*` group. So the bots you block stay blocked even though `*` allows everything. The `*` group applies to crawlers you did not name, such as Googlebot and Bingbot.

`User-agent: *` and `Disallow: /`.` https://example.com/robots.txt`. Each subdomain, such as `blog.example.com`, needs its own file.`/Blog/` and `/blog/` are different paths. Bot names are case-insensitive, but copy the vendor's spelling anyway.

``` python
from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
for bot in ["GPTBot", "OAI-SearchBot", "PerplexityBot"]:
    print(bot, rp.can_fetch(bot, "https://example.com/blog/post"))
```

Its wildcard handling differs from Google's, so verify key paths with a second parser.

Decide your policy first: allow all, block training only, or block everything. Write named groups for the bots you care about, host the file at the root, and verify it with logs and a parser.

If you would rather generate these groups than type them, the free generator at [https://citescore.vercel.app/robots-txt-ai-crawler-generator](https://citescore.vercel.app/robots-txt-ai-crawler-generator) builds them.