{"slug": "your-cdn-might-be-blocking-chatgpt-from-your-docs-and-robots-txt-won-t-show-it", "title": "Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It)", "summary": "A developer found that roughly 27% of websites unintentionally block major AI crawlers like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot at the CDN or WAF layer, even when robots.txt appears permissive. The developer advises checking edge logs for 403 errors and ensuring server-side rendering for AI crawlers that don't execute JavaScript. A live experiment tracking AI citation behavior is being published at connectingdots.live/experiment.", "body_md": "\n\n```\ncurl -sI -A \"GPTBot\" https://yoursite.com/ | head -1\n(Invoke-WebRequest -Uri \"https://yoursite.com/\" -UserAgent \"GPTBot\").StatusCode\ncurl -I -A \"GPTBot\" https://yoursite.com/\n```\n\nNote: Replace yoursite.com with your domain\n\nIf that returns anything other than 200, AI engines cannot read your site and your robots.txt may look perfectly permissive while it happens. Audits of several thousand sites found roughly 27% block at least one major AI crawler, most of them unintentionally, at the CDN or WAF layer rather than in robots.txt.\n\nBot-protection rules ship with sensible-sounding defaults: block unknown user agents, rate-limit non-browser traffic, challenge anything without JavaScript. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot all trip at least one of those on common configurations. Your marketing team sees a clean robots.txt; your edge quietly returns 403.\n\nfor UA in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot Bingbot; do\n\n`printf \"%-16s %s\\n\" \"$UA\" \"$(curl -sI -A \"$UA\" https://yoursite.com/ | head -1)\"`\n\ndone\n\nThen check your edge logs for 403s grouped by user agent that's where the truth lives.\n\nThe second failure: client-side rendering\n\nSeveral AI crawlers don't execute JavaScript. If your H1 and opening paragraph arrive via hydration, they see an empty shell. Server-render anything you want quoted view-source: is the test, not DevTools.\n\n``` python\nimport type { MetadataRoute } from 'next'\n\nexport default function robots(): MetadataRoute.Robots {\n\n  const disallow = ['/admin', '/api/']\n\n  return {\n\n    rules: [\n\n      { userAgent: '*', allow: '/', disallow },\n\n      { userAgent: 'GPTBot', allow: '/', disallow },\n\n      { userAgent: 'OAI-SearchBot', allow: '/', disallow },\n\n      { userAgent: 'ClaudeBot', allow: '/', disallow },\n\n      { userAgent: 'PerplexityBot', allow: '/', disallow },\n\n      { userAgent: 'Google-Extended', allow: '/', disallow },\n\n    ],\n\n    sitemap: 'https://yoursite.com/sitemap.xml',\n\n  }\n\n}\n```\n\nrobots.txt is necessary but not sufficient the CDN rule is the one that actually bites.\n\nIt's the precondition, not the strategy. Being retrieved is won with ordinary search ranking; being cited is won with extractable structure direct answers, explicit definitions, sourced statistics. But none of that runs if the crawler gets a 403 at the door.\n\nI'm publishing a live experiment on this: 20 fixed questions asked to ChatGPT and Gemini daily, with raw answers, cited-source leaderboards and the retrieved-but-not-cited gap published openly\n\n[connectingdots.live/experiment]", "url": "https://wpnews.pro/news/your-cdn-might-be-blocking-chatgpt-from-your-docs-and-robots-txt-won-t-show-it", "canonical_source": "https://dev.to/acetrondi/your-cdn-might-be-blocking-chatgpt-from-your-docs-and-robotstxt-wont-show-it-4e8k", "published_at": "2026-07-19 05:16:06+00:00", "updated_at": "2026-07-19 05:27:36.649190+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot", "ChatGPT", "Gemini", "connectingdots.live"], "alternates": {"html": "https://wpnews.pro/news/your-cdn-might-be-blocking-chatgpt-from-your-docs-and-robots-txt-won-t-show-it", "markdown": "https://wpnews.pro/news/your-cdn-might-be-blocking-chatgpt-from-your-docs-and-robots-txt-won-t-show-it.md", "text": "https://wpnews.pro/news/your-cdn-might-be-blocking-chatgpt-from-your-docs-and-robots-txt-won-t-show-it.txt", "jsonld": "https://wpnews.pro/news/your-cdn-might-be-blocking-chatgpt-from-your-docs-and-robots-txt-won-t-show-it.jsonld"}}