More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue link. Whether those answers can ever cite you is decided by one file: robots.txt
. And most robots.txt files decide it by accident.
| Kind | User agents | Blocking it means |
|---|---|---|
| Training | ||
GPTBot , ClaudeBot , Google-Extended , Applebot-Extended , CCBot |
||
| your content isn't used to train models | ||
| Retrieval | ||
OAI-SearchBot , ChatGPT-User , Claude-SearchBot , Claude-User , PerplexityBot |
||
| you can never be cited in an AI answer — no link, no attribution, ever |
Blocking training crawlers is a legitimate choice. Blocking retrieval agents is almost never what a site intended — it just silently removes you from AI answers.
The "block the AI bots" wave of 2024/25 produced a lot of this:
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
Disallow: /
That single group blocks training and retrieval — worst of both worlds if you still hold the training position but wanted visibility. I checked major news sites this week: nytimes.com allows 1 of 7 retrieval agents, reuters.com, theverge.com and wired.com 2 of 7 each. None of them is fully citable — and these are sites whose entire business is being the cited source.
A deliberate policy — citable, but not training material — looks like this:
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
Here's how I fell into this myself. My own site's served robots.txt contained a "managed" block list my CDN (Cloudflare) had injected above my hand-written rules — blocking the same crawlers my own section explicitly allowed, plus a Content-Signal:
line that is an express reservation of rights under Article 4 of the EU DSM Directive. A legal statement, published in my name, that I had never written.
Two lessons:
curl https://yoursite/robots.txt
and look for sections you didn't write.Allow
beats Disallow
on equal length, a group naming the agent beats *
) resolves conflicts curl
your page and check whether the text is in the HTML.FAQPage
markup.curl https://yoursite/robots.txt
Disallow
that hits them = you're out of AI answers.Content-Signal:
lines you didn't write.curl
a key page and check the text is present without JavaScript.For one site, that's honestly enough. I also built the bulk/scheduled version — a technical SEO audit that includes exactly this citability check per vendor, plus a diff against the previous run: SEO Audit + AI Visibility on Apify. Disclosure: that's my tool; the manual check above needs nothing but curl.