cd /news/ai-tools/16-of-32-well-known-sites-block-at-l… Β· home β€Ί topics β€Ί ai-tools β€Ί article
[ARTICLE Β· art-130207] src=ai-visibility.lastminutedealshq.com β†— pub= topic=ai-tools verified=true sentiment=Β· neutral

16 of 32 well-known sites block at least one AI search crawler

A check of 32 well-known websites' public robots.txt files on 7 September 2026 found that 16 block at least one AI search crawler, and 10 block both OAI-SearchBot, used by ChatGPT for citations, and PerplexityBot, used by Perplexity, making those sites uncitable by either engine. News publishers account for most of the blockers, a choice often tied to content licensing, while the remaining 6 sites block only Perplexity. The report also issued two corrections dated 7 September 2026: npr.org was initially listed as allowing both crawlers but actually blocks both, and figma.com was initially listed as blocking both but its robots.txt allows OAI-SearchBot and PerplexityBot on its homepage and a curated list of pages.

read4 min views4 publishedSep 15, 2026
16 of 32 well-known sites block at least one AI search crawler
Image: source

A check of 32 recognizable sites: 16 block at least one AI search crawler and 10 are invisible to both ChatGPT and Perplexity. Read from their public robots.txt, September 2026.

I ran the AI-visibility checker on 32 well-known sites to see how many let the AI search crawlers read them. 16 of the 32 block at least one. 10 block both the crawler ChatGPT uses for citations (OAI-SearchBot) and the one Perplexity uses (PerplexityBot), so those sites cannot be cited by either engine right now. Another 6 block only Perplexity.

News publishers make up most of the blockers, and for them it is often a deliberate choice tied to content licensing. This page does not judge that. It reads each site's public robots.txt and reports what the robots.txt allows. For a business that wants to show up in AI answers, a blocked search crawler means the same thing either way: no citations from that engine.

Each site links to its full result. Re-checked automatically, most recently on 7 September 2026. A question mark means the site's robots.txt could not be fetched on the last run, so the honest answer is that I do not know.

Tracking started 2026-09-07. Nothing on this list has changed its robots.txt since then. Any change will be listed here with the date it was first seen.

Correction. npr.org was first published here as allowing both crawlers. That was wrong. The checker could not fetch npr.org's robots.txt and treated the failure as permission. npr.org blocks OAI-SearchBot and PerplexityBot. Corrected 7 September 2026.

Correction. figma.com was first published here as blocking both crawlers. That was wrong. Figma's robots.txt allows OAI-SearchBot and PerplexityBot on its homepage and a curated list of pages using rules anchored with a dollar sign, such as "Allow: /$", and blocks the rest of the site. My matcher did not handle that anchor, so the allow rules were missed and the site read as fully blocked. Corrected 7 September 2026. The table reports homepage access, so figma now shows as reachable, with most of the rest of the site still disallowed.

| Site | Category | Can ChatGPT reach it? | Can Perplexity reach it? |

|---|---|---|---|
| [amazon.com](https://ai-visibility.lastminutedealshq.com/r/amazon.com) | Ecommerce | βœ— | βœ— | 
| [bbc.com](https://ai-visibility.lastminutedealshq.com/r/bbc.com) | News | βœ— | βœ— | 
| [bloomberg.com](https://ai-visibility.lastminutedealshq.com/r/bloomberg.com) | News | βœ— | βœ— | 
| [cnn.com](https://ai-visibility.lastminutedealshq.com/r/cnn.com) | News | βœ— | βœ— | 
| [npr.org](https://ai-visibility.lastminutedealshq.com/r/npr.org) | News | βœ— | βœ— | 
| [nytimes.com](https://ai-visibility.lastminutedealshq.com/r/nytimes.com) | News | βœ— | βœ— | 
| [pinterest.com](https://ai-visibility.lastminutedealshq.com/r/pinterest.com) | Media / Community | βœ— | βœ— | 
| [quora.com](https://ai-visibility.lastminutedealshq.com/r/quora.com) | Media / Community | βœ— | βœ— | 
| [reddit.com](https://ai-visibility.lastminutedealshq.com/r/reddit.com) | Media / Community | βœ— | βœ— | 
| [yelp.com](https://ai-visibility.lastminutedealshq.com/r/yelp.com) | Reference / Local | βœ— | βœ— | 
| [ebay.com](https://ai-visibility.lastminutedealshq.com/r/ebay.com) | Ecommerce | βœ“ | βœ— | 
| [forbes.com](https://ai-visibility.lastminutedealshq.com/r/forbes.com) | News | βœ“ | βœ— | 
| [linkedin.com](https://ai-visibility.lastminutedealshq.com/r/linkedin.com) | Reference / Local | βœ“ | βœ— | 
| [reuters.com](https://ai-visibility.lastminutedealshq.com/r/reuters.com) | News | βœ“ | βœ— | 
| [theguardian.com](https://ai-visibility.lastminutedealshq.com/r/theguardian.com) | News | βœ“ | βœ— | 
| [wsj.com](https://ai-visibility.lastminutedealshq.com/r/wsj.com) | News | βœ“ | βœ— | 
| [atlassian.com](https://ai-visibility.lastminutedealshq.com/r/atlassian.com) | SaaS / Tech | βœ“ | βœ“ | 
| [etsy.com](https://ai-visibility.lastminutedealshq.com/r/etsy.com) | Ecommerce | βœ“ | βœ“ | 
| [figma.com](https://ai-visibility.lastminutedealshq.com/r/figma.com) | SaaS / Tech | βœ“ | βœ“ | 
| [github.com](https://ai-visibility.lastminutedealshq.com/r/github.com) | SaaS / Tech | βœ“ | βœ“ | 
| [hubspot.com](https://ai-visibility.lastminutedealshq.com/r/hubspot.com) | SaaS / Tech | βœ“ | βœ“ | 
| [medium.com](https://ai-visibility.lastminutedealshq.com/r/medium.com) | Media / Community | βœ“ | βœ“ | 
| [notion.so](https://ai-visibility.lastminutedealshq.com/r/notion.so) | SaaS / Tech | βœ“ | βœ“ | 
| [openai.com](https://ai-visibility.lastminutedealshq.com/r/openai.com) | SaaS / Tech | βœ“ | βœ“ | 
| [salesforce.com](https://ai-visibility.lastminutedealshq.com/r/salesforce.com) | SaaS / Tech | βœ“ | βœ“ | 
| [shopify.com](https://ai-visibility.lastminutedealshq.com/r/shopify.com) | Ecommerce | βœ“ | βœ“ | 
| [stripe.com](https://ai-visibility.lastminutedealshq.com/r/stripe.com) | SaaS / Tech | βœ“ | βœ“ | 
| [substack.com](https://ai-visibility.lastminutedealshq.com/r/substack.com) | Media / Community | βœ“ | βœ“ | 
| [tripadvisor.com](https://ai-visibility.lastminutedealshq.com/r/tripadvisor.com) | Reference / Local | βœ“ | βœ“ | 
| [walmart.com](https://ai-visibility.lastminutedealshq.com/r/walmart.com) | Ecommerce | βœ“ | βœ“ | 
| [wayfair.com](https://ai-visibility.lastminutedealshq.com/r/wayfair.com) | Ecommerce | βœ“ | βœ“ | 
| [wikipedia.org](https://ai-visibility.lastminutedealshq.com/r/wikipedia.org) | Reference / Local | βœ“ | βœ“ | 

A green check means the site allows that engine's search crawler. A red mark means its robots.txt blocks it. ChatGPT Search uses OAI-SearchBot; Perplexity uses PerplexityBot.

This page covers 32 names most people recognise. The full list has every site in the census that blocks an AI search crawler, several hundred of them, each with its own result.

Want to know where your own site stands? Check it here, see the full 5,000-site crawler census, or run the same checks from the command line with aicheck.

── more in #ai-tools 4 stories Β· sorted by recency
── more on @chatgpt 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/16-of-32-well-known-…] indexed:0 read:4min 2026-09-15 Β· β€”