cd /news/ai-crawlers/cloudflare-block-ai-bots-what-to-blo… · home › topics › ai-crawlers › article
[ARTICLE · art-171755] src=dev.to ↗ pub= topic=ai-crawlers verified=true sentiment=· neutral

Cloudflare Block AI Bots: What to Block, What to Allow

A developer tested Cloudflare's AI bot controls by sending identical curl requests with different User-Agent strings to two Cloudflare-fronted sites, finding that a site blocking AI training crawlers returned 403 to GPTBot, ClaudeBot, CCBot and Bytespider while still returning 200 to OAI-SearchBot, ChatGPT-User, Claude-User and Googlebot. The test also showed the block was enforced at Cloudflare's edge rather than through robots.txt, and the developer cautioned that spoofed user agents only reveal how a rule treats a name, not how verified crawlers from OpenAI's or Anthropic's IP ranges are handled.

key takeaway

Blocking training crawlers like GPTBot, ClaudeBot, CCBot and Bytespider at Cloudflare's edge returned 403 while OAI-SearchBot, ChatGPT-User, Claude-User and Googlebot still got 200, so sites can opt out of training without disappearing from AI search answers.

by read8 min views2 publishedNov 18, 2026

Cloudflare will happily block AI bots for you. The question is which ones. "AI bot" covers three different jobs: crawlers that collect text to train future models, crawlers that build an AI search index, and fetchers that open one page because a person just asked ChatGPT or Claude about it. Block all three and you're out of the training sets, but you've also vanished from the AI answers that now send real visitors.

I ran the same request against two sites behind Cloudflare, changing only the bot name, to see what each setting actually does. Below are the results, the current dashboard path (it moved in 2026), and what I'd pick for a business site or a blog.

Each company runs separate crawlers for separate jobs. Blocking one does not block the others.

Every big AI company now publishes separate user agents, and their documentation spells out what each one is for:

GPTBot collects training data. OAI-SearchBot indexes pages for ChatGPT search. ChatGPT-User fetches a page when a user's request needs it, and because a person started it, OpenAI notes that robots.txt rules "may not apply". Changes to robots.txt take about 24 hours to be picked up.ClaudeBot is training, Claude-SearchBot improves search results, Claude-User fetches for a person using Claude. Anthropic says its bots respect robots.txt, including Crawl-delay. PerplexityBot builds its search index and, per Perplexity, isn't used to train foundation models. Perplexity-User "generally ignores robots.txt rules", because a user asked for the page.Google-Extended isn't a crawler at all. It's a robots.txt token that tells Google not to use your pages for Gemini training. It "does not impact a site's inclusion in Google Search". Normal Googlebot crawling is unaffected either way.CCBot (Common Crawl, a dataset many models train on), Bytespider (ByteDance), Amazonbot and meta-externalagent. All four are on Cloudflare's own managed robots.txt block list. The practical takeaway: "should I block AI bots?" is really three questions, and most sites answer them differently.

To see the difference, I sent the same request with curl to two sites behind Cloudflare and changed only the User-Agent. Site A is this blog, with no AI blocking. Site B is a company site that has AI training blocked.

18 November 2026. Same request, only the User-Agent changed. Training names are blocked on Site B; search and user agents pass.

On Site B the four training crawlers got a 403; everything else, including OAI-SearchBot, ChatGPT-User, Claude-User and Googlebot, got a 200. Site B's robots.txt has no AI rules at all, so the block happens at Cloudflare's edge, before the request reaches the server. That's the split you want: the training bots are refused, and the AI search engines can still read and cite the site.

One caveat that matters if you repeat this. My requests came from an ordinary connection, not from OpenAI's or Anthropic's IP ranges. Cloudflare identifies verified bots by where they come from, and its policies cover "Verified bots with that behavior and matching unverified bots". So a spoofed name tells you what the rule does with that name. It doesn't prove how the real crawler is treated. For that, look at the real traffic (next sections). You can run the same test on your own site:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://example.com/

for ua in GPTBot ClaudeBot CCBot Bytespider OAI-SearchBot ChatGPT-User \
          Claude-SearchBot Claude-User PerplexityBot Perplexity-User Googlebot; do
  printf "%-18s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/
done

The three policies and a sensible starting point for a site that wants to be cited by AI search.

Older guides tell you to flip a single Block AI bots switch. That switch has been replaced. In the dashboard today, pick the domain and go to Security › Settings, filter by Bot traffic, and open AI Crawler and Agent access policies. You set one action per behaviour:

"Block on pages with ads" is for sites that earn from display ads and don't want AI tools reading those pages without showing the ads. For a business site with no ads it behaves like Allow.

If you remember Cloudflare announcing that it blocks AI crawlers by default: that applied to the earlier rollout. The current docs say new domains start with Search, Agent and Training all on "Allow (do not block)". Don't assume you're protected. Check the setting, or run the curl test above.

Blocking at the edge and robots.txt are two separate layers. Cloudflare can also manage robots.txt for you (all plans). With it on, Cloudflare puts its own block in front of your file: a Content-signal line saying search is fine and AI training isn't (search=yes, ai-train=no), followed by Disallow rules for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent.

Content Signals are a newer, voluntary format with three values: search, ai-input (using your page to answer a question live) and ai-train. Nothing forces a crawler to honour them, the same as the rest of robots.txt, which is why the edge block exists. If you'd rather write robots.txt by hand, this is the equivalent:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Check it with a robots.txt validator before you deploy: a stray Disallow: / under User-agent: * blocks Googlebot as well.

If the three groups are too coarse, AI Crawl Control (it used to be called AI Audit) lists each crawler that visited your site, with its request count and how often it ignored robots.txt. Each row has an action: Allow, Block, or Charge (pay per crawl, still a closed beta). Block creates a WAF custom rule for you. It's available on every plan, with two limits on Free: detection is by User-Agent only, and you can't customise the block response.

This is the screen to use for the verified-bot question above. Look at what really arrives, not what a spoofed curl request shows. It's also where you'd allow one mixed-purpose crawler that the Training policy caught.

This is the other half of the search traffic: owners who never meant to block anything and find that ChatGPT says it can't open their page. Work through it in this order:

Disallow for OAI-SearchBot or Claude-SearchBot, or a blanket one under *. curl: if the text you care about isn't in the response, the bot can't see it either. Run the curl loop from earlier with your own domain. A 403 or a challenge page for OAI-SearchBot or ChatGPT-User is the answer. A 200 with your content in it means the block is somewhere else, or the AI tool simply hasn't crawled you yet.

Whatever you choose, it's a setting, not a one-off decision. Look at AI Crawl Control after a month and adjust. If Cloudflare also caches your HTML, why cf-cache-status says DYNAMIC and does Cloudflare cache 404 responses cover the caching side of the same dashboard.

Should I block AI bots on Cloudflare?

Block AI training crawlers if you don't want your content used to train future models, but allow AI search crawlers and user-requested fetchers if you want to appear in ChatGPT, Claude and Perplexity answers. In Cloudflare's AI Crawler and Agent access policies that means Training: Block, Search: Allow, Agent: Allow.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot indexes pages for ChatGPT search. They are controlled separately: blocking GPTBot keeps you out of training, while blocking OAI-SearchBot removes your site from ChatGPT search answers. ChatGPT-User is a third agent that fetches a page when a user asks for it.

Does blocking AI bots hurt my Google rankings?

No. Blocking Google-Extended, GPTBot or ClaudeBot does not affect normal Google Search; Google states that Google-Extended is not a ranking signal and does not change inclusion in Search. Only blocking Googlebot would. Check that a robots.txt rule meant for AI bots does not sit under User-agent: *.

Does Cloudflare block AI bots by default?

Not on new domains, according to Cloudflare's current docs: Search, Agent and Training all start on Allow (do not block). Check Security, Settings, filter Bot traffic, AI Crawler and Agent access policies, or send a test request with curl using an AI bot's User-Agent.

Why can't ChatGPT access my website?

Common causes are Cloudflare's Search or Agent policy set to Block, Bot Fight Mode or a WAF rule catching the fetcher, a JavaScript challenge, a robots.txt Disallow for OAI-SearchBot, or content that only appears after JavaScript runs. Security Events in Cloudflare shows which rule refused the request.

Is Cloudflare AI Crawl Control free?

Yes, AI Crawl Control is available on all Cloudflare plans. It lists crawlers with their request counts and lets you allow or block each one. On the Free plan detection is by User-Agent only and the block response cannot be customised; pay per crawl is a closed beta.

Do AI bots respect robots.txt?

The major training and search crawlers say they do: OpenAI, Anthropic and Perplexity all document robots.txt support for GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. User-initiated fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it. To enforce a block, use Cloudflare's edge policies.

Originally published on knowledgemarkg.com.

── more in #ai-crawlers 4 stories · sorted by recency
── more on @cloudflare 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/cloudflare-block-ai-…] indexed:0 read:8min 2026-11-18 · —