16 of 32 well-known sites block at least one AI search crawler

A check of 32 well-known websites' public robots.txt files on 7 September 2026 found that 16 block at least one AI search crawler, and 10 block both OAI-SearchBot, used by ChatGPT for citations, and PerplexityBot, used by Perplexity, making those sites uncitable by either engine. News publishers account for most of the blockers, a choice often tied to content licensing, while the remaining 6 sites block only Perplexity. The report also issued two corrections dated 7 September 2026: npr.org was initially listed as allowing both crawlers but actually blocks both, and figma.com was initially listed as blocking both but its robots.txt allows OAI-SearchBot and PerplexityBot on its homepage and a curated list of pages.

Which well-known sites block AI search crawlers A check of 32 recognizable sites: 16 block at least one AI search crawler and 10 are invisible to both ChatGPT and Perplexity. Read from their public robots.txt, September 2026. I ran the AI-visibility checker on 32 well-known sites to see how many let the AI search crawlers read them. 16 of the 32 block at least one. 10 block both the crawler ChatGPT uses for citations OAI-SearchBot and the one Perplexity uses PerplexityBot , so those sites cannot be cited by either engine right now. Another 6 block only Perplexity. News publishers make up most of the blockers, and for them it is often a deliberate choice tied to content licensing. This page does not judge that. It reads each site's public robots.txt and reports what the robots.txt allows. For a business that wants to show up in AI answers, a blocked search crawler means the same thing either way: no citations from that engine. Each site links to its full result. Re-checked automatically, most recently on 7 September 2026. A question mark means the site's robots.txt could not be fetched on the last run, so the honest answer is that I do not know. Tracking started 2026-09-07. Nothing on this list has changed its robots.txt since then. Any change will be listed here with the date it was first seen. Correction. npr.org was first published here as allowing both crawlers. That was wrong. The checker could not fetch npr.org's robots.txt and treated the failure as permission. npr.org blocks OAI-SearchBot and PerplexityBot. Corrected 7 September 2026. Correction. figma.com was first published here as blocking both crawlers. That was wrong. Figma's robots.txt allows OAI-SearchBot and PerplexityBot on its homepage and a curated list of pages using rules anchored with a dollar sign, such as "Allow: /$", and blocks the rest of the site. My matcher did not handle that anchor, so the allow rules were missed and the site read as fully blocked. Corrected 7 September 2026. The table reports homepage access, so figma now shows as reachable, with most of the rest of the site still disallowed. | Site | Category | Can ChatGPT reach it? | Can Perplexity reach it? | |---|---|---|---| | amazon.com https://ai-visibility.lastminutedealshq.com/r/amazon.com | Ecommerce | ✗ | ✗ | | bbc.com https://ai-visibility.lastminutedealshq.com/r/bbc.com | News | ✗ | ✗ | | bloomberg.com https://ai-visibility.lastminutedealshq.com/r/bloomberg.com | News | ✗ | ✗ | | cnn.com https://ai-visibility.lastminutedealshq.com/r/cnn.com | News | ✗ | ✗ | | npr.org https://ai-visibility.lastminutedealshq.com/r/npr.org | News | ✗ | ✗ | | nytimes.com https://ai-visibility.lastminutedealshq.com/r/nytimes.com | News | ✗ | ✗ | | pinterest.com https://ai-visibility.lastminutedealshq.com/r/pinterest.com | Media / Community | ✗ | ✗ | | quora.com https://ai-visibility.lastminutedealshq.com/r/quora.com | Media / Community | ✗ | ✗ | | reddit.com https://ai-visibility.lastminutedealshq.com/r/reddit.com | Media / Community | ✗ | ✗ | | yelp.com https://ai-visibility.lastminutedealshq.com/r/yelp.com | Reference / Local | ✗ | ✗ | | ebay.com https://ai-visibility.lastminutedealshq.com/r/ebay.com | Ecommerce | ✓ | ✗ | | forbes.com https://ai-visibility.lastminutedealshq.com/r/forbes.com | News | ✓ | ✗ | | linkedin.com https://ai-visibility.lastminutedealshq.com/r/linkedin.com | Reference / Local | ✓ | ✗ | | reuters.com https://ai-visibility.lastminutedealshq.com/r/reuters.com | News | ✓ | ✗ | | theguardian.com https://ai-visibility.lastminutedealshq.com/r/theguardian.com | News | ✓ | ✗ | | wsj.com https://ai-visibility.lastminutedealshq.com/r/wsj.com | News | ✓ | ✗ | | atlassian.com https://ai-visibility.lastminutedealshq.com/r/atlassian.com | SaaS / Tech | ✓ | ✓ | | etsy.com https://ai-visibility.lastminutedealshq.com/r/etsy.com | Ecommerce | ✓ | ✓ | | figma.com https://ai-visibility.lastminutedealshq.com/r/figma.com | SaaS / Tech | ✓ | ✓ | | github.com https://ai-visibility.lastminutedealshq.com/r/github.com | SaaS / Tech | ✓ | ✓ | | hubspot.com https://ai-visibility.lastminutedealshq.com/r/hubspot.com | SaaS / Tech | ✓ | ✓ | | medium.com https://ai-visibility.lastminutedealshq.com/r/medium.com | Media / Community | ✓ | ✓ | | notion.so https://ai-visibility.lastminutedealshq.com/r/notion.so | SaaS / Tech | ✓ | ✓ | | openai.com https://ai-visibility.lastminutedealshq.com/r/openai.com | SaaS / Tech | ✓ | ✓ | | salesforce.com https://ai-visibility.lastminutedealshq.com/r/salesforce.com | SaaS / Tech | ✓ | ✓ | | shopify.com https://ai-visibility.lastminutedealshq.com/r/shopify.com | Ecommerce | ✓ | ✓ | | stripe.com https://ai-visibility.lastminutedealshq.com/r/stripe.com | SaaS / Tech | ✓ | ✓ | | substack.com https://ai-visibility.lastminutedealshq.com/r/substack.com | Media / Community | ✓ | ✓ | | tripadvisor.com https://ai-visibility.lastminutedealshq.com/r/tripadvisor.com | Reference / Local | ✓ | ✓ | | walmart.com https://ai-visibility.lastminutedealshq.com/r/walmart.com | Ecommerce | ✓ | ✓ | | wayfair.com https://ai-visibility.lastminutedealshq.com/r/wayfair.com | Ecommerce | ✓ | ✓ | | wikipedia.org https://ai-visibility.lastminutedealshq.com/r/wikipedia.org | Reference / Local | ✓ | ✓ | A green check means the site allows that engine's search crawler. A red mark means its robots.txt blocks it. ChatGPT Search uses OAI-SearchBot; Perplexity uses PerplexityBot. This page covers 32 names most people recognise. The full list https://ai-visibility.lastminutedealshq.com/sites-blocking-ai-search has every site in the census that blocks an AI search crawler, several hundred of them, each with its own result. Want to know where your own site stands? Check it here https://ai-visibility.lastminutedealshq.com/ , see the full 5,000-site crawler census https://ai-visibility.lastminutedealshq.com/data , or run the same checks from the command line with aicheck https://ai-visibility.lastminutedealshq.com/tools/aicheck.py .