Your robots.txt decides whether ChatGPT can cite you. Here's the 5-minute check.

A developer warns that robots.txt files often accidentally block AI retrieval agents, preventing sites from being cited in ChatGPT, Claude, and Perplexity answers. The developer found that major news sites like nytimes.com, reuters.com, theverge.com, and wired.com allow only a fraction of retrieval agents, and discovered that Cloudflare had injected rules into their own robots.txt. They recommend a simple curl check to ensure retrieval agents are allowed and offer a tool for automated audits.

More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue link. Whether those answers can ever cite you is decided by one file: robots.txt . And most robots.txt files decide it by accident. | Kind | User agents | Blocking it means | |---|---|---| Training | GPTBot , ClaudeBot , Google-Extended , Applebot-Extended , CCBot | your content isn't used to train models | Retrieval | OAI-SearchBot , ChatGPT-User , Claude-SearchBot , Claude-User , PerplexityBot | you can never be cited in an AI answer — no link, no attribution, ever | Blocking training crawlers is a legitimate choice. Blocking retrieval agents is almost never what a site intended — it just silently removes you from AI answers. The "block the AI bots" wave of 2024/25 produced a lot of this: User-agent: GPTBot User-agent: ChatGPT-User User-agent: ClaudeBot Disallow: / That single group blocks training and retrieval — worst of both worlds if you still hold the training position but wanted visibility. I checked major news sites this week: nytimes.com allows 1 of 7 retrieval agents, reuters.com, theverge.com and wired.com 2 of 7 each. None of them is fully citable — and these are sites whose entire business is being the cited source. A deliberate policy — citable, but not training material — looks like this: User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / Here's how I fell into this myself. My own site's served robots.txt contained a "managed" block list my CDN Cloudflare had injected above my hand-written rules — blocking the same crawlers my own section explicitly allowed, plus a Content-Signal: line that is an express reservation of rights under Article 4 of the EU DSM Directive. A legal statement, published in my name, that I had never written. Two lessons: curl https://yoursite/robots.txt and look for sections you didn't write. Allow beats Disallow on equal length, a group naming the agent beats resolves conflicts curl your page and check whether the text is in the HTML. FAQPage markup. curl https://yoursite/robots.txt Disallow that hits them = you're out of AI answers. Content-Signal: lines you didn't write. curl a key page and check the text is present without JavaScript.For one site, that's honestly enough. I also built the bulk/scheduled version — a technical SEO audit that includes exactly this citability check per vendor, plus a diff against the previous run: SEO Audit + AI Visibility on Apify https://apify.com/siccscha/seo-health-auditor . Disclosure: that's my tool; the manual check above needs nothing but curl.