Originally published on pagelens.io. Most advice about AI search starts with "check your robots.txt". I wanted to know what the biggest sites actually do, so I ran an AI visibility check on the top 1,500 domains of the Tranco list (list 4672X), a research ranking of the most visited domains, on October 12, 2026.
Many of those domains are APIs and CDNs with no home page (gstatic.com, akamai.net and so on). 740 served a home page to a normal browser request, and every number below is out of those 740 websites.
Share of the 740 websites whose robots.txt keeps each agent off the home page (in practice, off the site):
| Agent | What it does | Blocked |
|---|---|---|
| CCBot | Common Crawl, an open dataset many AI labs train on | 16.8% |
| Bytespider | ByteDance training crawler | 15.7% |
| GPTBot | OpenAI training crawler | 14.7% |
| ClaudeBot | Anthropic training crawler | 13.6% |
| Google-Extended | Gemini training (a robots.txt token, not a separate crawler) | 13.2% |
| meta-externalagent | Meta training crawler | 12.6% |
| PerplexityBot | Perplexity search index | 10.9% |
| ChatGPT-User | ChatGPT opening a page for a person | 9.7% |
| Applebot-Extended | Apple Intelligence training | 9.3% |
| Perplexity-User | Perplexity opening a page for a person | 8.4% |
| Claude-User | Claude opening a page for a person | 8.0% |
| OAI-SearchBot | ChatGPT search index | 7.8% |
Most of these are deliberate: 92 of the 109 GPTBot blocks name GPTBot explicitly, the rest come from a rule for all bots.
GPTBot only collects training data. Whether a site appears in ChatGPT search depends on OAI-SearchBot, and whether ChatGPT can open a page while answering someone depends on ChatGPT-User. OpenAI documents them as separate agents with separate robots.txt rules.
Of the 109 websites that block GPTBot:
Both are legitimate choices. The second one is what most businesses want: stay out of training, stay in the answers. If you copied a "block AI bots" robots.txt from a blog post, check which group you're in.
For each site I also requested the home page with each bot's name in the user agent (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot) and compared the answer with a normal browser request. On 130 sites, at least one bot-named request was turned away (a 403, a challenge page or no answer) although robots.txt allows that bot. I then re-tested those 130 sites with a fourth request: a generic, made-up crawler name. That splits them into groups:
One caution: these requests don't come from OpenAI's or Anthropic's servers. Some CDNs block requests that claim to be GPTBot but don't come from the IP ranges OpenAI publishes, and let the real GPTBot in. From the outside, "blocks AI bots" and "blocks impostors" look the same, which is why I don't name sites in this group. If you're one of them, your CDN's bot settings (in Cloudflare: Security → Bots) will tell you which it is.
The useful lesson for a site owner: robots.txt can say "allowed" while the server says no, and nothing in robots.txt will show it.
AI crawlers don't run JavaScript. On 22% of home pages, fewer than 120 words were readable in the HTML before scripts run. This is an upper bound: some sites send bots a short interstitial page instead of the real one. But for a single-page app, an empty HTML shell is what every AI crawler sees.
| Signal | Share of the 740 sites |
|---|---|
| Has a robots.txt | 76% |
| Has a sitemap (in robots.txt or at /sitemap.xml) | 62% |
| Has structured data (JSON-LD) on the home page | 45% |
| Missing a title or meta description | 21% |
| Publishes an llms.txt (Markdown with a title) | 14% |
llms.txt is still a minority practice even among the largest sites: about one in seven. Several more sites answered /llms.txt with something that isn't one (an IP address, a bare "OK", an error page), which a naive check would count.
If you want to run the same check on your site, I built it as a free tool (no sign-up): pagelens.io/tools/ai-crawler-checker. Happy to answer questions about the data or the method in the comments.