# Your robots.txt is probably blocking the wrong AI bots. Here's how to sort training, search and user fetchers

> Source: <https://dev.to/sigzenai/your-robotstxt-is-probably-blocking-the-wrong-ai-bots-heres-how-to-sort-training-search-and-d51>
> Published: 2026-11-16 08:49:34+00:00

**TL;DR:** "Block AI bots" is three different decisions, not one. Training crawlers, AI search crawlers and user-triggered fetchers each do a different job. Block the wrong one and your product disappears from ChatGPT and Google's AI answers, while your content still ends up in training data anyway.

For years, `robots.txt` was a set-and-forget file. Then AI crawlers arrived, and someone on the marketing or legal side asked engineering to "block the AI bots".

The traffic numbers explain the pressure. Cloudflare's [July 2026 crawler report](https://blog.cloudflare.com/agentic-internet-bot-report/) found that **52% of crawler requests on its network were for AI training in June 2026**, up from 22% in spring 2025. More than half of all web traffic is now non-human.

(Note that Cloudflare sells bot-management products and argues for blocking by default, and the figures cover only its own network. They are still the best public view we have.)

The catch: the quick fix, a blanket `Disallow` for anything that looks like AI, usually blocks the bots that put you *into* AI answers.

Before touching the file, label every AI user agent by what it actually does:

| Type | What it does | Example user agents | If you block it | 
|---|---|---|---|
| **Training** | Collects pages to train future models | `GPTBot` | Your content is (per the vendor) not used for training. AI answers are unaffected in the short term. | 
| **Search / answer** | Indexes pages so the assistant can cite them in live answers | `OAI-SearchBot` | You stop appearing in ChatGPT search answers | 
| **User-triggered** | Fetches a page when a user asks the assistant about it | `ChatGPT-User` | Inconsistent: OpenAI notes that robots.txt rules may not apply to these fetches | 

Google works differently, and this is where most teams slip up:

`Googlebot`` Google-Extended`
Vendors rename and add agents regularly. Check the official lists, such as [OpenAI's crawler docs](https://platform.openai.com/docs/bots) and [Google's crawler overview](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), before you edit anything.

If you're a business whose buyers use ChatGPT or Gemini to research vendors, you usually want to be *found* in AI answers. A reasonable starting point:

```
# --- AI search / answer crawlers: allow (keeps you in live AI answers)
User-agent: OAI-SearchBot
Allow: /

# --- User-triggered fetchers: allow (a real person asked about your page)
User-agent: ChatGPT-User
Allow: /

# --- Training crawlers: a business decision (see below)
# Option A: allow (your content can shape how future models describe you)
User-agent: GPTBot
Allow: /
# Option B: block training only
# User-agent: GPTBot
# Disallow: /

# --- Google: never block Googlebot if you want Search + AI Overviews
User-agent: Googlebot
Allow: /

# Google-Extended: opt-out token for some AI uses; does NOT affect AI Overviews
# User-agent: Google-Extended
# Disallow: /

# Keep private areas private for everyone
User-agent: *
Disallow: /admin/
Disallow: /api/
```

Treat this as a template, not gospel. Publishers whose content *is* the product (news, research, courses) have a very different calculation from a SaaS or services company.

This is the step most audits miss: **your CDN or WAF can override robots.txt.**

If Cloudflare, Akamai, Fastly or your WAF has a "block AI bots" toggle switched on, your carefully written `Allow` lines do nothing. The bot never reaches the file. So check both layers:

A quick way to see what a bot experiences is to request a page with its user agent and check the status code:

```
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://example.com/
```

A `403` or a challenge page means something upstream is blocking it, whatever robots.txt says. This isn't a perfect test: some CDNs verify real bots by IP address, so a spoofed user agent can be treated differently. It does catch the most common misconfiguration, a blanket bot block.

If you want a quick read on where you stand, the team at **[Sigzen AI](https://sigzenai.com/insights/block-ai-training-crawlers-keep-answer-bots)** has written up the full decision table behind this post, including what happens with each block/allow combination.

They also built a free [AI crawler check](https://sigzenai.com/free-tools/ai-crawler-check). Enter a domain and it reads your robots.txt against 16 named AI crawlers (OpenAI, Google, Microsoft, Perplexity, Anthropic, Apple, Amazon, Meta, ByteDance, Common Crawl and others), returning *allowed*, *blocked* or *partly blocked* for each. It only reads robots.txt, so pair it with the `curl` test above to catch CDN-level blocks.

*How does your team handle this? Do you block training bots, allow everything, or let the CDN decide? I'd like to hear in the comments.*

*Written by the Sigzen AI editorial team. Sigzen AI helps B2B brands get described accurately in ChatGPT, Gemini, Copilot and Google AI answers, and builds AI agents for operations teams. More of our research is at [sigzenai.com/insights](https://sigzenai.com/insights).*
