# Is Cloudflare Blocking ChatGPT From Your Site? Check in 30 Seconds

> Source: <https://dev.to/xwickz/is-cloudflare-blocking-chatgpt-from-your-site-check-in-30-seconds-3imi>
> Published: 2026-11-20 17:36:00+00:00

**Key takeaway:** Of 286 audited portfolios, 19 blocked GPTBot and ClaudeBot and 17 of those blocks came from Cloudflare's managed robots.txt, so check whether your training-crawler blocks were deliberate and test AI search crawlers with a curl user-agent request to catch firewall-level 403s.

**Short answer:** probably not ChatGPT *search*, but probably ChatGPT *training*. Cloudflare's managed `robots.txt` blocks training crawlers like GPTBot and ClaudeBot. It doesn't block OAI-SearchBot or ChatGPT-User, which fetch pages to answer and cite. Open `yourdomain.com/robots.txt` to check the first part, and run the `curl` below to check the second.

When I audited 286 developer portfolios for [an earlier study](https://wickz.dev/en/blog/portfolios-invisible-to-ai/), 19 blocked GPTBot and ClaudeBot. When I opened those files, **17 of the 19 blocks had been written by Cloudflare**, not by the owner. Most of those people never decided to block AI. A dashboard setting decided for them.

This is the part most posts get wrong. "AI bots" aren't one thing:

| Crawler | Company | What it does | 
|---|---|---|
| GPTBot | OpenAI | Collects content to **train** models | 
| OAI-SearchBot | OpenAI | Indexes pages for **ChatGPT search** | 
| ChatGPT-User | OpenAI | Fetches a page **when a user asks** about it | 
| ClaudeBot | Anthropic | Collects content to **train** models | 
| Claude-SearchBot | Anthropic | Indexes pages for **Claude search** | 
| PerplexityBot | Perplexity | Indexes pages for **Perplexity answers** | 
| Google-Extended |  | Controls whether Gemini **trains** on your content (Google Search isn't affected) | 

Blocking a training crawler keeps you out of future models. Blocking a search crawler keeps you out of the answers. Those are very different outcomes.

Open `https://yourdomain.com/robots.txt` in your browser. If Cloudflare manages it, you'll see something like this at the top:

```
# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes,ai-train=no
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
# (Amazonbot, Applebot-Extended, Bytespider, CCBot, meta-externalagent...)
# END Cloudflare Managed Content
```

That's the training list. Search crawlers aren't in it, so with only this block, ChatGPT search and Perplexity can still read you. The `Content-signal` line says the same thing in words: search yes, training no.

If you see `Disallow: /` under `OAI-SearchBot`, `ChatGPT-User` or `PerplexityBot`, that one *is* blocking you from AI answers. Someone added it by hand or with a plugin.

`robots.txt` is only a request. Cloudflare also has **AI Crawl Control**, which blocks crawlers at the firewall with a `403`. That doesn't show up in `robots.txt` at all, so you have to ask as the bot:

```
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot/1.0; +https://openai.com/searchbot" https://yourdomain.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://yourdomain.com/
```

If the second line prints `200` and the first prints `403` (or `402`), you're blocked at the firewall. Swap in `ChatGPT-User`, `PerplexityBot` or `GPTBot` to test the others. On Windows, use `curl.exe` and `NUL` instead of `/dev/null`.

This isn't perfect. Cloudflare can also verify real bots by IP, so a fake user agent from your laptop won't always get the same answer as the real crawler. But a `403` here is a strong hint to go look at the dashboard.

At the time of writing, both settings live in the Cloudflare dashboard for your domain:

My take for a portfolio or a small business site: **allow the search crawlers**, because that's how ChatGPT, Claude and Perplexity find and recommend you. Training is a real choice, and blocking it is reasonable. Just make it on purpose.

To allow everything explicitly in a `robots.txt` you write yourself:

```
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
```

[Run your site through the free audit](https://audit.wickz.dev/en/?utm_source=devto&utm_campaign=cloudflare-blocking-chatgpt). It reads your `robots.txt` and tells you which AI search crawlers and which training crawlers are blocked, separately, in about ten seconds and with no sign-up.
