# Your robots.txt decides whether ChatGPT can cite you. Here's the 5-minute check.

> Source: <https://dev.to/siccscha/your-robotstxt-decides-whether-chatgpt-can-cite-you-heres-the-5-minute-check-185l>
> Published: 2026-07-31 22:23:09+00:00

More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue link. Whether those answers can ever cite *you* is decided by one file: `robots.txt`

. And most robots.txt files decide it by accident.

| Kind | User agents | Blocking it means |
|---|---|---|
Training |
`GPTBot` , `ClaudeBot` , `Google-Extended` , `Applebot-Extended` , `CCBot`
|
your content isn't used to train models |
Retrieval |
`OAI-SearchBot` , `ChatGPT-User` , `Claude-SearchBot` , `Claude-User` , `PerplexityBot`
|
you can never be cited in an AI answer — no link, no attribution, ever |

Blocking training crawlers is a legitimate choice. Blocking retrieval agents is almost never what a site *intended* — it just silently removes you from AI answers.

The "block the AI bots" wave of 2024/25 produced a lot of this:

```
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
Disallow: /
```

That single group blocks training **and** retrieval — worst of both worlds if you still hold the training position but wanted visibility. I checked major news sites this week: nytimes.com allows **1 of 7** retrieval agents, reuters.com, theverge.com and wired.com **2 of 7** each. None of them is fully citable — and these are sites whose entire business is being the cited source.

A deliberate policy — citable, but not training material — looks like this:

```
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
```

Here's how I fell into this myself. My own site's served robots.txt contained a "managed" block list my CDN (Cloudflare) had injected **above** my hand-written rules — blocking the same crawlers my own section explicitly allowed, plus a `Content-Signal:`

line that is an express reservation of rights under Article 4 of the EU DSM Directive. A legal statement, published in my name, that I had never written.

Two lessons:

`curl https://yoursite/robots.txt`

and look for sections you didn't write.`Allow`

beats `Disallow`

on equal length, a group naming the agent beats `*`

) resolves conflicts `curl`

your page and check whether the text is in the HTML.`FAQPage`

markup.`curl https://yoursite/robots.txt`

`Disallow`

that hits them = you're out of AI answers.`Content-Signal:`

lines you didn't write.`curl`

a key page and check the text is present without JavaScript.For one site, that's honestly enough. I also built the bulk/scheduled version — a technical SEO audit that includes exactly this citability check per vendor, plus a diff against the previous run: [SEO Audit + AI Visibility on Apify](https://apify.com/siccscha/seo-health-auditor). Disclosure: that's my tool; the manual check above needs nothing but curl.