{"slug": "is-cloudflare-blocking-chatgpt-from-your-site-check-in-30-seconds", "title": "Is Cloudflare Blocking ChatGPT From Your Site? Check in 30 Seconds", "summary": "A developer audited 286 developer portfolios and found that 19 blocked GPTBot and ClaudeBot training crawlers, with 17 of those blocks written by Cloudflare's managed robots.txt rather than by site owners. The writeup distinguishes training crawlers (GPTBot, ClaudeBot, Google-Extended) from AI search crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) and offers robots.txt inspection plus curl user-agent tests to detect firewall-level blocks such as Cloudflare's AI Crawl Control returning 403s.", "body_md": "**Short answer:** probably not ChatGPT *search*, but probably ChatGPT *training*. Cloudflare's managed `robots.txt` blocks training crawlers like GPTBot and ClaudeBot. It doesn't block OAI-SearchBot or ChatGPT-User, which fetch pages to answer and cite. Open `yourdomain.com/robots.txt` to check the first part, and run the `curl` below to check the second.\n\nWhen I audited 286 developer portfolios for [an earlier study](https://wickz.dev/en/blog/portfolios-invisible-to-ai/), 19 blocked GPTBot and ClaudeBot. When I opened those files, **17 of the 19 blocks had been written by Cloudflare**, not by the owner. Most of those people never decided to block AI. A dashboard setting decided for them.\n\nThis is the part most posts get wrong. \"AI bots\" aren't one thing:\n\n| Crawler | Company | What it does | \n|---|---|---|\n| GPTBot | OpenAI | Collects content to **train** models | \n| OAI-SearchBot | OpenAI | Indexes pages for **ChatGPT search** | \n| ChatGPT-User | OpenAI | Fetches a page **when a user asks** about it | \n| ClaudeBot | Anthropic | Collects content to **train** models | \n| Claude-SearchBot | Anthropic | Indexes pages for **Claude search** | \n| PerplexityBot | Perplexity | Indexes pages for **Perplexity answers** | \n| Google-Extended |  | Controls whether Gemini **trains** on your content (Google Search isn't affected) | \n\nBlocking a training crawler keeps you out of future models. Blocking a search crawler keeps you out of the answers. Those are very different outcomes.\n\nOpen `https://yourdomain.com/robots.txt` in your browser. If Cloudflare manages it, you'll see something like this at the top:\n\n```\n# BEGIN Cloudflare Managed content\nUser-Agent: *\nContent-signal: search=yes,ai-train=no\nAllow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n# (Amazonbot, Applebot-Extended, Bytespider, CCBot, meta-externalagent...)\n# END Cloudflare Managed Content\n```\n\nThat's the training list. Search crawlers aren't in it, so with only this block, ChatGPT search and Perplexity can still read you. The `Content-signal` line says the same thing in words: search yes, training no.\n\nIf you see `Disallow: /` under `OAI-SearchBot`, `ChatGPT-User` or `PerplexityBot`, that one *is* blocking you from AI answers. Someone added it by hand or with a plugin.\n\n`robots.txt` is only a request. Cloudflare also has **AI Crawl Control**, which blocks crawlers at the firewall with a `403`. That doesn't show up in `robots.txt` at all, so you have to ask as the bot:\n\n```\ncurl -s -o /dev/null -w \"%{http_code}\\n\" -A \"OAI-SearchBot/1.0; +https://openai.com/searchbot\" https://yourdomain.com/\ncurl -s -o /dev/null -w \"%{http_code}\\n\" -A \"Mozilla/5.0\" https://yourdomain.com/\n```\n\nIf the second line prints `200` and the first prints `403` (or `402`), you're blocked at the firewall. Swap in `ChatGPT-User`, `PerplexityBot` or `GPTBot` to test the others. On Windows, use `curl.exe` and `NUL` instead of `/dev/null`.\n\nThis isn't perfect. Cloudflare can also verify real bots by IP, so a fake user agent from your laptop won't always get the same answer as the real crawler. But a `403` here is a strong hint to go look at the dashboard.\n\nAt the time of writing, both settings live in the Cloudflare dashboard for your domain:\n\nMy take for a portfolio or a small business site: **allow the search crawlers**, because that's how ChatGPT, Claude and Perplexity find and recommend you. Training is a real choice, and blocking it is reasonable. Just make it on purpose.\n\nTo allow everything explicitly in a `robots.txt` you write yourself:\n\n```\nUser-agent: OAI-SearchBot\nUser-agent: ChatGPT-User\nUser-agent: Claude-SearchBot\nUser-agent: PerplexityBot\nAllow: /\n```\n\n[Run your site through the free audit](https://audit.wickz.dev/en/?utm_source=devto&utm_campaign=cloudflare-blocking-chatgpt). It reads your `robots.txt` and tells you which AI search crawlers and which training crawlers are blocked, separately, in about ten seconds and with no sign-up.", "url": "https://wpnews.pro/news/is-cloudflare-blocking-chatgpt-from-your-site-check-in-30-seconds", "canonical_source": "https://dev.to/xwickz/is-cloudflare-blocking-chatgpt-from-your-site-check-in-30-seconds-3imi", "published_at": "2026-11-20 17:36:00+00:00", "updated_at": "2026-11-20 17:43:54.517314+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "agent-protocols"], "entities": ["Cloudflare", "OpenAI", "Anthropic", "Perplexity", "Google", "GPTBot", "ClaudeBot", "OAI-SearchBot"], "also_reported_by": [], "takeaway": "Of 286 audited portfolios, 19 blocked GPTBot and ClaudeBot and 17 of those blocks came from Cloudflare's managed robots.txt, so check whether your training-crawler blocks were deliberate and test AI search crawlers with a curl user-agent request to catch firewall-level 403s.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/is-cloudflare-blocking-chatgpt-from-your-site-check-in-30-seconds", "markdown": "https://wpnews.pro/news/is-cloudflare-blocking-chatgpt-from-your-site-check-in-30-seconds.md", "text": "https://wpnews.pro/news/is-cloudflare-blocking-chatgpt-from-your-site-check-in-30-seconds.txt", "jsonld": "https://wpnews.pro/news/is-cloudflare-blocking-chatgpt-from-your-site-check-in-30-seconds.jsonld"}}