cd /news/developer-tools/your-site-might-rank-on-google-but-s… · home › topics › developer-tools › article
[ARTICLE · art-65054] src=dev.to ↗ pub= topic=developer-tools verified=true sentiment=· neutral

Your site might rank on Google but still be invisible to ChatGPT. Learn how to test GPTBot, ClaudeBot, PerplexityBot, Bingbot, and other crawlers, detect hidden CDN blocks, and make your content…

A developer discovered that CDN configurations, not robots.txt, often block AI crawlers like GPTBot and ClaudeBot from accessing documentation. Testing revealed that Cloudflare's security settings can silently block these bots, making content invisible to ChatGPT even when Google indexes it.

read1 min views28 publishedJul 19, 2026

Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It) acetrondi acetrondi acetrondi Follow Jul 19 Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It)

#

webdev

#

seo

#

ai

#

nextjs 5 reactions 1 comment 2 min read

source & further reading
dev.to — original article I build Apify Actors for RAG pipelines — before/after: one Actor turns 41K of HTML soup into 3.6K of clean Markdown The Zero-Effort Local File Janitor I Built a Menu Bar App So I'd Stop Pasting Slack Messages Into ChatGPT
── more in #developer-tools 4 stories · sorted by recency
── more on @gptbot 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/your-site-might-rank…] indexed:0 read:1min 2026-07-19 · —