{"slug": "your-robots-txt-decides-whether-chatgpt-can-cite-you-here-s-the-5-minute-check", "title": "Your robots.txt decides whether ChatGPT can cite you. Here's the 5-minute check.", "summary": "A developer warns that robots.txt files often accidentally block AI retrieval agents, preventing sites from being cited in ChatGPT, Claude, and Perplexity answers. The developer found that major news sites like nytimes.com, reuters.com, theverge.com, and wired.com allow only a fraction of retrieval agents, and discovered that Cloudflare had injected rules into their own robots.txt. They recommend a simple curl check to ensure retrieval agents are allowed and offer a tool for automated audits.", "body_md": "More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue link. Whether those answers can ever cite *you* is decided by one file: `robots.txt`\n\n. And most robots.txt files decide it by accident.\n\n| Kind | User agents | Blocking it means |\n|---|---|---|\nTraining |\n`GPTBot` , `ClaudeBot` , `Google-Extended` , `Applebot-Extended` , `CCBot`\n|\nyour content isn't used to train models |\nRetrieval |\n`OAI-SearchBot` , `ChatGPT-User` , `Claude-SearchBot` , `Claude-User` , `PerplexityBot`\n|\nyou can never be cited in an AI answer — no link, no attribution, ever |\n\nBlocking training crawlers is a legitimate choice. Blocking retrieval agents is almost never what a site *intended* — it just silently removes you from AI answers.\n\nThe \"block the AI bots\" wave of 2024/25 produced a lot of this:\n\n```\nUser-agent: GPTBot\nUser-agent: ChatGPT-User\nUser-agent: ClaudeBot\nDisallow: /\n```\n\nThat single group blocks training **and** retrieval — worst of both worlds if you still hold the training position but wanted visibility. I checked major news sites this week: nytimes.com allows **1 of 7** retrieval agents, reuters.com, theverge.com and wired.com **2 of 7** each. None of them is fully citable — and these are sites whose entire business is being the cited source.\n\nA deliberate policy — citable, but not training material — looks like this:\n\n```\nUser-agent: OAI-SearchBot\nUser-agent: ChatGPT-User\nUser-agent: Claude-SearchBot\nUser-agent: Claude-User\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: Google-Extended\nUser-agent: CCBot\nDisallow: /\n```\n\nHere's how I fell into this myself. My own site's served robots.txt contained a \"managed\" block list my CDN (Cloudflare) had injected **above** my hand-written rules — blocking the same crawlers my own section explicitly allowed, plus a `Content-Signal:`\n\nline that is an express reservation of rights under Article 4 of the EU DSM Directive. A legal statement, published in my name, that I had never written.\n\nTwo lessons:\n\n`curl https://yoursite/robots.txt`\n\nand look for sections you didn't write.`Allow`\n\nbeats `Disallow`\n\non equal length, a group naming the agent beats `*`\n\n) resolves conflicts `curl`\n\nyour page and check whether the text is in the HTML.`FAQPage`\n\nmarkup.`curl https://yoursite/robots.txt`\n\n`Disallow`\n\nthat hits them = you're out of AI answers.`Content-Signal:`\n\nlines you didn't write.`curl`\n\na key page and check the text is present without JavaScript.For one site, that's honestly enough. I also built the bulk/scheduled version — a technical SEO audit that includes exactly this citability check per vendor, plus a diff against the previous run: [SEO Audit + AI Visibility on Apify](https://apify.com/siccscha/seo-health-auditor). Disclosure: that's my tool; the manual check above needs nothing but curl.", "url": "https://wpnews.pro/news/your-robots-txt-decides-whether-chatgpt-can-cite-you-here-s-the-5-minute-check", "canonical_source": "https://dev.to/siccscha/your-robotstxt-decides-whether-chatgpt-can-cite-you-heres-the-5-minute-check-185l", "published_at": "2026-07-31 22:23:09+00:00", "updated_at": "2026-07-31 22:40:43.303547+00:00", "lang": "en", "topics": ["ai-products", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["ChatGPT", "Claude", "Perplexity", "Cloudflare", "nytimes.com", "reuters.com", "theverge.com", "wired.com"], "alternates": {"html": "https://wpnews.pro/news/your-robots-txt-decides-whether-chatgpt-can-cite-you-here-s-the-5-minute-check", "markdown": "https://wpnews.pro/news/your-robots-txt-decides-whether-chatgpt-can-cite-you-here-s-the-5-minute-check.md", "text": "https://wpnews.pro/news/your-robots-txt-decides-whether-chatgpt-can-cite-you-here-s-the-5-minute-check.txt", "jsonld": "https://wpnews.pro/news/your-robots-txt-decides-whether-chatgpt-can-cite-you-here-s-the-5-minute-check.jsonld"}}