Your robots.txt is probably blocking the wrong AI bots. Here's how to sort training, search and user fetchers A developer's guide argues that blocking AI crawlers via robots.txt is three separate decisions—training crawlers like GPTBot, AI search crawlers like OAI-SearchBot, and user-triggered fetchers like ChatGPT-User—and that a blanket Disallow typically removes a site from live AI answers while leaving training exposure largely unchanged. The guide, citing Cloudflare's July 2026 crawler report that 52% of crawler requests on its network were for AI training in June 2026, recommends allowing search and user-triggered agents, treating training crawlers as a business decision, and never blocking Googlebot, while warning that CDN or WAF 'block AI bots' toggles can override robots.txt entirely. TL;DR: "Block AI bots" is three different decisions, not one. Training crawlers, AI search crawlers and user-triggered fetchers each do a different job. Block the wrong one and your product disappears from ChatGPT and Google's AI answers, while your content still ends up in training data anyway. For years, robots.txt was a set-and-forget file. Then AI crawlers arrived, and someone on the marketing or legal side asked engineering to "block the AI bots". The traffic numbers explain the pressure. Cloudflare's July 2026 crawler report https://blog.cloudflare.com/agentic-internet-bot-report/ found that 52% of crawler requests on its network were for AI training in June 2026 , up from 22% in spring 2025. More than half of all web traffic is now non-human. Note that Cloudflare sells bot-management products and argues for blocking by default, and the figures cover only its own network. They are still the best public view we have. The catch: the quick fix, a blanket Disallow for anything that looks like AI, usually blocks the bots that put you into AI answers. Before touching the file, label every AI user agent by what it actually does: | Type | What it does | Example user agents | If you block it | |---|---|---|---| | Training | Collects pages to train future models | GPTBot | Your content is per the vendor not used for training. AI answers are unaffected in the short term. | | Search / answer | Indexes pages so the assistant can cite them in live answers | OAI-SearchBot | You stop appearing in ChatGPT search answers | | User-triggered | Fetches a page when a user asks the assistant about it | ChatGPT-User | Inconsistent: OpenAI notes that robots.txt rules may not apply to these fetches | Google works differently, and this is where most teams slip up: Googlebot Google-Extended Vendors rename and add agents regularly. Check the official lists, such as OpenAI's crawler docs https://platform.openai.com/docs/bots and Google's crawler overview https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers , before you edit anything. If you're a business whose buyers use ChatGPT or Gemini to research vendors, you usually want to be found in AI answers. A reasonable starting point: --- AI search / answer crawlers: allow keeps you in live AI answers User-agent: OAI-SearchBot Allow: / --- User-triggered fetchers: allow a real person asked about your page User-agent: ChatGPT-User Allow: / --- Training crawlers: a business decision see below Option A: allow your content can shape how future models describe you User-agent: GPTBot Allow: / Option B: block training only User-agent: GPTBot Disallow: / --- Google: never block Googlebot if you want Search + AI Overviews User-agent: Googlebot Allow: / Google-Extended: opt-out token for some AI uses; does NOT affect AI Overviews User-agent: Google-Extended Disallow: / Keep private areas private for everyone User-agent: Disallow: /admin/ Disallow: /api/ Treat this as a template, not gospel. Publishers whose content is the product news, research, courses have a very different calculation from a SaaS or services company. This is the step most audits miss: your CDN or WAF can override robots.txt. If Cloudflare, Akamai, Fastly or your WAF has a "block AI bots" toggle switched on, your carefully written Allow lines do nothing. The bot never reaches the file. So check both layers: A quick way to see what a bot experiences is to request a page with its user agent and check the status code: curl -s -o /dev/null -w "%{http code}\n" \ -A "Mozilla/5.0 compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot " \ https://example.com/ A 403 or a challenge page means something upstream is blocking it, whatever robots.txt says. This isn't a perfect test: some CDNs verify real bots by IP address, so a spoofed user agent can be treated differently. It does catch the most common misconfiguration, a blanket bot block. If you want a quick read on where you stand, the team at Sigzen AI https://sigzenai.com/insights/block-ai-training-crawlers-keep-answer-bots has written up the full decision table behind this post, including what happens with each block/allow combination. They also built a free AI crawler check https://sigzenai.com/free-tools/ai-crawler-check . Enter a domain and it reads your robots.txt against 16 named AI crawlers OpenAI, Google, Microsoft, Perplexity, Anthropic, Apple, Amazon, Meta, ByteDance, Common Crawl and others , returning allowed , blocked or partly blocked for each. It only reads robots.txt, so pair it with the curl test above to catch CDN-level blocks. How does your team handle this? Do you block training bots, allow everything, or let the CDN decide? I'd like to hear in the comments. Written by the Sigzen AI editorial team. Sigzen AI helps B2B brands get described accurately in ChatGPT, Gemini, Copilot and Google AI answers, and builds AI agents for operations teams. More of our research is at sigzenai.com/insights https://sigzenai.com/insights .