curl -sI -A "GPTBot" https://yoursite.com/ | head -1
(Invoke-WebRequest -Uri "https://yoursite.com/" -UserAgent "GPTBot").StatusCode
curl -I -A "GPTBot" https://yoursite.com/
Note: Replace yoursite.com with your domain
If that returns anything other than 200, AI engines cannot read your site and your robots.txt may look perfectly permissive while it happens. Audits of several thousand sites found roughly 27% block at least one major AI crawler, most of them unintentionally, at the CDN or WAF layer rather than in robots.txt.
Bot-protection rules ship with sensible-sounding defaults: block unknown user agents, rate-limit non-browser traffic, challenge anything without JavaScript. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot all trip at least one of those on common configurations. Your marketing team sees a clean robots.txt; your edge quietly returns 403.
for UA in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot Bingbot; do
printf "%-16s %s\n" "$UA" "$(curl -sI -A "$UA" https://yoursite.com/ | head -1)"
done
Then check your edge logs for 403s grouped by user agent that's where the truth lives.
The second failure: client-side rendering
Several AI crawlers don't execute JavaScript. If your H1 and opening paragraph arrive via hydration, they see an empty shell. Server-render anything you want quoted view-source: is the test, not DevTools.
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
const disallow = ['/admin', '/api/']
return {
rules: [
{ userAgent: '*', allow: '/', disallow },
{ userAgent: 'GPTBot', allow: '/', disallow },
{ userAgent: 'OAI-SearchBot', allow: '/', disallow },
{ userAgent: 'ClaudeBot', allow: '/', disallow },
{ userAgent: 'PerplexityBot', allow: '/', disallow },
{ userAgent: 'Google-Extended', allow: '/', disallow },
],
sitemap: 'https://yoursite.com/sitemap.xml',
}
}
robots.txt is necessary but not sufficient the CDN rule is the one that actually bites.
It's the precondition, not the strategy. Being retrieved is won with ordinary search ranking; being cited is won with extractable structure direct answers, explicit definitions, sourced statistics. But none of that runs if the crawler gets a 403 at the door.
I'm publishing a live experiment on this: 20 fixed questions asked to ChatGPT and Gemini daily, with raw answers, cited-source leaderboards and the retrieved-but-not-cited gap published openly
[connectingdots.live/experiment]