{"slug": "we-checked-740-of-the-world-s-top-websites-for-ai-bots-most-that-block-gptbot", "title": "We checked 740 of the world's top websites for AI bots. Most that block GPTBot also block ChatGPT search", "summary": "A developer analyzed the robots.txt and server behavior of 740 of the top 1,500 Tranco-ranked domains, finding that 14.7% block OpenAI's training crawler GPTBot while only 7.8% block OAI-SearchBot, the agent that controls ChatGPT search indexing. The study also found that 130 sites returned 403s or challenge pages to bot-named requests despite robots.txt allowing them, and that 22% of home pages expose fewer than 120 words of readable HTML before JavaScript runs.", "body_md": "*Originally published on [pagelens.io](https://pagelens.io/blog/ai-bots-top-websites-study).*\n\nMost advice about AI search starts with \"check your robots.txt\". I wanted to know what the biggest sites actually do, so I ran an AI visibility check on the top 1,500 domains of the [Tranco list](https://tranco-list.eu/) (list 4672X), a research ranking of the most visited domains, on October 12, 2026.\n\nMany of those domains are APIs and CDNs with no home page (gstatic.com, akamai.net and so on). **740 served a home page to a normal browser request**, and every number below is out of those 740 websites.\n\nShare of the 740 websites whose robots.txt keeps each agent off the home page (in practice, off the site):\n\n| Agent | What it does | Blocked | \n|---|---|---|\n| CCBot | Common Crawl, an open dataset many AI labs train on | 16.8% | \n| Bytespider | ByteDance training crawler | 15.7% | \n| GPTBot | OpenAI training crawler | 14.7% | \n| ClaudeBot | Anthropic training crawler | 13.6% | \n| Google-Extended | Gemini training (a robots.txt token, not a separate crawler) | 13.2% | \n| meta-externalagent | Meta training crawler | 12.6% | \n| PerplexityBot | Perplexity search index | 10.9% | \n| ChatGPT-User | ChatGPT opening a page for a person | 9.7% | \n| Applebot-Extended | Apple Intelligence training | 9.3% | \n| Perplexity-User | Perplexity opening a page for a person | 8.4% | \n| Claude-User | Claude opening a page for a person | 8.0% | \n| OAI-SearchBot | ChatGPT search index | 7.8% | \n\nMost of these are deliberate: 92 of the 109 GPTBot blocks name GPTBot explicitly, the rest come from a rule for all bots.\n\nGPTBot only collects training data. Whether a site appears in ChatGPT search depends on **OAI-SearchBot**, and whether ChatGPT can open a page while answering someone depends on **ChatGPT-User**. OpenAI documents them as separate agents with separate robots.txt rules.\n\nOf the 109 websites that block GPTBot:\n\nBoth are legitimate choices. The second one is what most businesses want: stay out of training, stay in the answers. If you copied a \"block AI bots\" robots.txt from a blog post, check which group you're in.\n\nFor each site I also requested the home page with each bot's name in the user agent (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot) and compared the answer with a normal browser request. On 130 sites, at least one bot-named request was turned away (a 403, a challenge page or no answer) although robots.txt allows that bot.\n\nI then re-tested those 130 sites with a fourth request: a generic, made-up crawler name. That splits them into groups:\n\nOne caution: these requests don't come from OpenAI's or Anthropic's servers. Some CDNs block requests that *claim* to be GPTBot but don't come from the IP ranges OpenAI publishes, and let the real GPTBot in. From the outside, \"blocks AI bots\" and \"blocks impostors\" look the same, which is why I don't name sites in this group. If you're one of them, your CDN's bot settings (in Cloudflare: Security → Bots) will tell you which it is.\n\nThe useful lesson for a site owner: robots.txt can say \"allowed\" while the server says no, and nothing in robots.txt will show it.\n\nAI crawlers don't run JavaScript. On **22%** of home pages, fewer than 120 words were readable in the HTML before scripts run. This is an upper bound: some sites send bots a short interstitial page instead of the real one. But for a single-page app, an empty HTML shell is what every AI crawler sees.\n\n| Signal | Share of the 740 sites | \n|---|---|\n| Has a robots.txt | 76% | \n| Has a sitemap (in robots.txt or at /sitemap.xml) | 62% | \n| Has structured data (JSON-LD) on the home page | 45% | \n| Missing a title or meta description | 21% | \n| Publishes an llms.txt (Markdown with a title) | 14% | \n\nllms.txt is still a minority practice even among the largest sites: about one in seven. Several more sites answered /llms.txt with something that isn't one (an IP address, a bare \"OK\", an error page), which a naive check would count.\n\nIf you want to run the same check on your site, I built it as a free tool (no sign-up): [pagelens.io/tools/ai-crawler-checker](https://pagelens.io/tools/ai-crawler-checker).\n\nHappy to answer questions about the data or the method in the comments.", "url": "https://wpnews.pro/news/we-checked-740-of-the-world-s-top-websites-for-ai-bots-most-that-block-gptbot", "canonical_source": "https://dev.to/ali_baizhanov_1c2dfd2f333/we-checked-740-of-the-worlds-top-websites-for-ai-bots-most-that-block-gptbot-also-block-chatgpt-463d", "published_at": "2026-10-12 08:27:34+00:00", "updated_at": "2026-10-12 08:40:21.313057+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-search", "structured-data", "agent-protocols", "generative-engine-optimization"], "entities": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "PerplexityBot", "CCBot", "OpenAI", "Tranco"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-checked-740-of-the-world-s-top-websites-for-ai-bots-most-that-block-gptbot", "markdown": "https://wpnews.pro/news/we-checked-740-of-the-world-s-top-websites-for-ai-bots-most-that-block-gptbot.md", "text": "https://wpnews.pro/news/we-checked-740-of-the-world-s-top-websites-for-ai-bots-most-that-block-gptbot.txt", "jsonld": "https://wpnews.pro/news/we-checked-740-of-the-world-s-top-websites-for-ai-bots-most-that-block-gptbot.jsonld"}}