cd /news/ai-crawlers/your-robots-txt-is-probably-blocking… · home › topics › ai-crawlers › article
[ARTICLE · art-170437] src=dev.to ↗ pub= topic=ai-crawlers verified=true sentiment=· neutral

Your robots.txt is probably blocking the wrong AI bots. Here's how to sort training, search and user fetchers

A developer's guide argues that blocking AI crawlers via robots.txt is three separate decisions—training crawlers like GPTBot, AI search crawlers like OAI-SearchBot, and user-triggered fetchers like ChatGPT-User—and that a blanket Disallow typically removes a site from live AI answers while leaving training exposure largely unchanged. The guide, citing Cloudflare's July 2026 crawler report that 52% of crawler requests on its network were for AI training in June 2026, recommends allowing search and user-triggered agents, treating training crawlers as a business decision, and never blocking Googlebot, while warning that CDN or WAF 'block AI bots' toggles can override robots.txt entirely.

by read4 min views2 publishedNov 16, 2026

TL;DR: "Block AI bots" is three different decisions, not one. Training crawlers, AI search crawlers and user-triggered fetchers each do a different job. Block the wrong one and your product disappears from ChatGPT and Google's AI answers, while your content still ends up in training data anyway.

For years, robots.txt was a set-and-forget file. Then AI crawlers arrived, and someone on the marketing or legal side asked engineering to "block the AI bots".

The traffic numbers explain the pressure. Cloudflare's July 2026 crawler report found that 52% of crawler requests on its network were for AI training in June 2026, up from 22% in spring 2025. More than half of all web traffic is now non-human.

(Note that Cloudflare sells bot-management products and argues for blocking by default, and the figures cover only its own network. They are still the best public view we have.)

The catch: the quick fix, a blanket Disallow for anything that looks like AI, usually blocks the bots that put you into AI answers.

Before touching the file, label every AI user agent by what it actually does:

Type What it does Example user agents If you block it
Training Collects pages to train future models GPTBot Your content is (per the vendor) not used for training. AI answers are unaffected in the short term.
Search / answer Indexes pages so the assistant can cite them in live answers OAI-SearchBot You stop appearing in ChatGPT search answers
User-triggered Fetches a page when a user asks the assistant about it ChatGPT-User Inconsistent: OpenAI notes that robots.txt rules may not apply to these fetches

Google works differently, and this is where most teams slip up:

Googlebot`` Google-Extended Vendors rename and add agents regularly. Check the official lists, such as OpenAI's crawler docs and Google's crawler overview, before you edit anything.

If you're a business whose buyers use ChatGPT or Gemini to research vendors, you usually want to be found in AI answers. A reasonable starting point:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: Googlebot
Allow: /


User-agent: *
Disallow: /admin/
Disallow: /api/

Treat this as a template, not gospel. Publishers whose content is the product (news, research, courses) have a very different calculation from a SaaS or services company.

This is the step most audits miss: your CDN or WAF can override robots.txt.

If Cloudflare, Akamai, Fastly or your WAF has a "block AI bots" toggle switched on, your carefully written Allow lines do nothing. The bot never reaches the file. So check both layers:

A quick way to see what a bot experiences is to request a page with its user agent and check the status code:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://example.com/

A 403 or a challenge page means something upstream is blocking it, whatever robots.txt says. This isn't a perfect test: some CDNs verify real bots by IP address, so a spoofed user agent can be treated differently. It does catch the most common misconfiguration, a blanket bot block.

If you want a quick read on where you stand, the team at Sigzen AI has written up the full decision table behind this post, including what happens with each block/allow combination.

They also built a free AI crawler check. Enter a domain and it reads your robots.txt against 16 named AI crawlers (OpenAI, Google, Microsoft, Perplexity, Anthropic, Apple, Amazon, Meta, ByteDance, Common Crawl and others), returning allowed, blocked or partly blocked for each. It only reads robots.txt, so pair it with the curl test above to catch CDN-level blocks.

How does your team handle this? Do you block training bots, allow everything, or let the CDN decide? I'd like to hear in the comments.

Written by the Sigzen AI editorial team. Sigzen AI helps B2B brands get described accurately in ChatGPT, Gemini, Copilot and Google AI answers, and builds AI agents for operations teams. More of our research is at sigzenai.com/insights.

── more in #ai-crawlers 4 stories · sorted by recency
── more on @cloudflare 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/your-robots-txt-is-p…] indexed:0 read:4min 2026-11-16 · —