I checked whether ChatGPT can cite the top 50 news sites. 38 are invisible — most by accident.

An automated audit of the 50 largest English-language news and tech publishers found that 38 are not citable by AI retrieval agents like ChatGPT, with 9 sites blocking all seven major retrieval bots and 25 blocking retrieval while allowing training crawlers. The audit, run by developer siccscha on August 1, 2026, revealed that many publishers inadvertently block citations due to robots.txt misconfigurations or JavaScript-dependent pages, while PerplexityBot is the most blocked agent and OpenAI's OAI-SearchBot is the least, reflecting licensing deals.

Three weeks ago I wrote about the difference between AI training crawlers and AI retrieval agents https://dev.to/siccscha/your-robotstxt-decides-whether-chatgpt-can-cite-you-heres-the-5-minute-check-185l : GPTBot collects data to train models, but it's ChatGPT-User and OAI-SearchBot that fetchA commenter said they hadn't realized these were different bots. That made me wonder how many professional publishers haven't either. So I measured it. On August 1, 2026 I ran an automated check against the 50 biggest English-language news and tech publishers. For each site: robots.txt and evaluate it for OAI-SearchBot , ChatGPT-User , Claude-SearchBot , Claude-User , PerplexityBot , Perplexity-User , Amazonbot and 8 training crawlers GPTBot , ClaudeBot , CCBot , Google-Extended , Applebot-Extended , Bytespider , meta-externalagent , anthropic-ai — using Google's documented robots semantics most specific group wins, longest rule wins, Allow wins ties , for path / .A site counts as citable only if both hold: no retrieval agent blocked, and actual text in the served HTML. The tool is an open audit Actor I built https://apify.com/siccscha/seo-health-auditor ; four sites NYT, Guardian, FT, Daily 38 of 50 are not citable. 9 sites block all seven retrieval agents: CNN, NBC News, USA Today, HuffPost, The Telegraph, Daily Mail, CNET, ZDNet, Mashable. Whatever these newsrooms publish, no AI answer can ever quote it or link to it. 25 sites block in the wrong direction. They allow at least one training crawler while blocking retrieval agents — meaning their content may train models, but the one thing that sends readers back a citation with a link is off. The Verge, Wired, Ars Technica, The Atlantic, Vox, The Guardian, The Washington Post and WSJ are all in this group. I doubt a single one chose that trade on purpose. 5 sites fail by exactly one agent. ABC News and TechCrunch block only ChatGPT-User ; Axios, Tom's Hardware and VentureBeat block only Amazonbot . One robots.txt line away from citable. 3 sites have the door open and the room empty. NPR, Politico and The Information allow all seven retrieval agents — and serve a homepage whose HTML contains essentially no readable text without JavaScript. Retrieval agents don't run JavaScript. A browser shows a normal page; an AI agent gets nothing to quote. This failure is invisible in every browser-based audit. Who gets blocked most tells its own story. PerplexityBot is blocked by 30 of 50 sites, Amazonbot by 28, Anthropic's two retrieval agents by 24–25 — but OpenAI's OAI-SearchBot by only 14. That's the licensing-deal era in one number: publishers with OpenAI deals let OpenAI's citation bot in and block everyone else's. The 12 citable sites: Fox News, CBS News, Business Insider, LA Times, Time, Slate, The Independent, The Daily Beast, Semafor, Engadget, Gizmodo, PCMag. Retrieval = citation agents allowed of 7 . Training = training crawlers allowed of 8 . = robots.txt only homepage bot-walled . | Site | Retrieval | Training | Citable | Why not | |---|---|---|---|---| | cnet.com | 0/7 | 1/8 | ❌ | robots.txt | | cnn.com | 0/7 | 1/8 | ❌ | robots.txt | | dailymail.co.uk | 0/7 | 0/8 | ❌ | robots.txt | | huffpost.com | 0/7 | 0/8 | ❌ | robots.txt | | mashable.com | 0/7 | 1/8 | ❌ | robots.txt | | nbcnews.com | 0/7 | 0/8 | ❌ | robots.txt | | telegraph.co.uk | 0/7 | 0/8 | ❌ | robots.txt | | usatoday.com | 0/7 | 0/8 | ❌ | robots.txt | | zdnet.com | 0/7 | 1/8 | ❌ | robots.txt | | bloomberg.com | 1/7 | 0/8 | ❌ | robots.txt | | economist.com | 1/7 | 1/8 | ❌ | robots.txt | | nytimes.com | 1/7 | 0/8 | ❌ | robots.txt | | arstechnica.com | 2/7 | 1/8 | ❌ | robots.txt | | bbc.com | 2/7 | 0/8 | ❌ | robots.txt | | cnbc.com | 2/7 | 0/8 | ❌ | robots.txt | | marketwatch.com | 2/7 | 1/8 | ❌ | robots.txt | | newyorker.com | 2/7 | 2/8 | ❌ | robots.txt | | reuters.com | 2/7 | 0/8 | ❌ | robots.txt | | theatlantic.com | 2/7 | 1/8 | ❌ | robots.txt | | theverge.com | 2/7 | 1/8 | ❌ | robots.txt | | vox.com | 2/7 | 1/8 | ❌ | robots.txt | | wired.com | 2/7 | 2/8 | ❌ | robots.txt | | wsj.com | 2/7 | 1/8 | ❌ | robots.txt | | apnews.com | 3/7 | 3/8 | ❌ | robots.txt | | nypost.com | 3/7 | 1/8 | ❌ | robots.txt | | theguardian.com | 3/7 | 2/8 | ❌ | robots.txt | | newsweek.com | 4/7 | 2/8 | ❌ | robots.txt | | forbes.com | 5/7 | 1/8 | ❌ | robots.txt | | ft.com | 5/7 | 1/8 | ❌ | robots.txt | | washingtonpost.com | 5/7 | 2/8 | ❌ | robots.txt | | abcnews.go.com | 6/7 | 3/8 | ❌ | blocks only ChatGPT-User | | axios.com | 6/7 | 6/8 | ❌ | blocks only Amazonbot | | techcrunch.com | 6/7 | 1/8 | ❌ | blocks only ChatGPT-User | | tomshardware.com | 6/7 | 6/8 | ❌ | blocks only Amazonbot | | venturebeat.com | 6/7 | 5/8 | ❌ | blocks only Amazonbot | | npr.org | 7/7 | 8/8 | ❌ | no text without JavaScript | | politico.com | 7/7 | 8/8 | ❌ | no text without JavaScript | | theinformation.com | 7/7 | 8/8 | ❌ | no text without JavaScript | | businessinsider.com | 7/7 | 4/8 | ✅ | | | cbsnews.com | 7/7 | 7/8 | ✅ | | | engadget.com | 7/7 | 8/8 | ✅ | | | foxnews.com | 7/7 | 8/8 | ✅ | | | gizmodo.com | 7/7 | 5/8 | ✅ | | | independent.co.uk | 7/7 | 8/8 | ✅ | | | latimes.com | 7/7 | 5/8 | ✅ | | | pcmag.com | 7/7 | 8/8 | ✅ | | | semafor.com | 7/7 | 7/8 | ✅ | | | slate.com | 7/7 | 8/8 | ✅ | | | thedailybeast.com | 7/7 | 8/8 | ✅ | | | time.com | 7/7 | 8/8 | ✅ | / and the homepage. Sections may differ.Two failure modes, both invisible in a browser and in classic SEO tools: OAI-SearchBot , ChatGPT-User , Claude-SearchBot , Claude-User , PerplexityBot , Perplexity-User , Amazonbot . Blanket AI-blocklists and one-click CDN blockers usually hit these too. The curl your page and look for your content in the HTML. If it only appears after JavaScript runs, retrieval agents see an empty shell.For a single site you honestly don't need a tool — the manual check takes five minutes. Where it stops being trivial: That's what I built SEO Health Auditor https://apify.com/siccscha/seo-health-auditor for: it runs both checks plus regular technical SEO across any list of sites, from $0.05 per page, and on a schedule it diffs against the previous run — so you learn about the drift before your traffic does. Raw data for all 50 sites available on request — happy to share the JSON.