05:16
2026-07-19
dev.to
large-language-models
Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It)
A developer found that roughly 27% of websites unintentionally block major AI crawlers like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot at the CDN or WAF layer, even when robots.txt appears pe…