Key findings #
- 29.8% of the robots.txt files that block paths for everyone and name an AI crawler let that
crawler past at least one of those blocks (329 of 1,105). In 182 of them, the
crawler's group only allows everything: a blanket
Allow: /or an emptyDisallow:. - The paths most often let back in: site search (155 files), login and account pages (136), APIs (92), admin areas (79) and cart or checkout pages (70).
- The crawlers let through most often: GPTBot (187), OAI-SearchBot (179), ChatGPT-User (178), PerplexityBot (150), ClaudeBot (139).
- 23.2% of robots.txt files name at least one AI crawler. GPTBot is blocked from the whole site by14.8% (21.6% among the top 1,000 domains); CCBot (15.3%) and Bytespider (15.1%) are blocked most often.
- 20.0% block at least one AI training crawler, but only 7.1% do so while leaving all 9 AI answer engines open.
- 14.4% of the sites that answered serve a valid llms.txt; 2.7% of robots.txt files
carry Cloudflare's
Content-Signalline. - WordPress sites name AI crawlers less often than other sites (16.9% against 22.0%).
The rule behind it #
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and lists the rules for
those crawlers. The standard, RFC 9309, has a crawler
look for the group that names it. Only if there is none does it fall back:
“If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.”
RFC 9309, section 2.2.1
Google documents the same thing for its own crawlers: user agent specific groups and the global * group
are not combined. So the moment a
file adds a group for GPTBot — even one that only says Allow: / to welcome it — GPTBot stops reading every rule
written under User-agent: *.
User-agent: *
Disallow: /search
Disallow: /account/
Disallow: /api/
User-agent: GPTBot
Allow: /
| Path | Every other crawlere.g. Googlebot | GPTBothas its own group |
|---|---|---|
/search?q=… |
Blocked | Allowed |
/account/settings |
Blocked | Allowed |
/api/v2/orders |
Blocked | Allowed |
/blog/any-post |
Allowed | Allowed |
Highlighted rows: paths the file blocks for everyone except the crawler it names.
Two ways to fix it
- If the group only exists to welcome the crawler (it says nothing but
Allow: /), delete it. The crawler falls back toUser-agent: *and follows the same rules as everyone else. - If the crawler needs rules of its own , repeat every
Disallowline fromUser-agent: *inside its group:
User-agent: GPTBot
Disallow: /search
Disallow: /account/
Disallow: /api/
Our free robots.txt checker finds this for you: paste a file and it lists every path a named crawler can
reach that User-agent: * blocks.
How we count it: a file has a gap when a path is blocked under User-agent: * and a named AI crawler's own group has
no rule for that path at all, or only a blanket Allow: /. If the group allows a specific path on purpose (for
example Allow: /api/public/), we don't count it. Blocking the whole site under * while allowing named
crawlers is a common deliberate pattern, so Disallow: / on its own doesn't count either.
What gets let back in #
None of these paths is secret — robots.txt is public, and it isn't access control. But they are the paths site owners block on purpose: search results that spawn endless near-duplicate URLs, account pages, and endpoints that cost money to serve.
| Path type | Share of files | Files |
|---|---|---|
| Site search results | 155 | |
| Login, account and sign-up pages | 136 | |
| API endpoints | 92 | |
| Admin areas | 79 | |
| Cart, checkout and orders | 70 | |
| Feeds and XML files | 44 |
Grouped by patterns in the blocked path (search, ?s=, login, account, /api/, admin, cart, checkout, feed and similar), so the types are approximate.
The most common shape is a group that only allows everything — a blanket Allow: / or an empty Disallow:: 182 of
the 329 files (55.3%). The other 147 give the crawler
rules of its own but leave out some of the paths blocked for everyone else.
| Crawler | Files | Operator |
|---|---|---|
GPTBot |
OpenAI | |
OAI-SearchBot |
OpenAI | |
ChatGPT-User |
OpenAI | |
PerplexityBot |
Perplexity | |
ClaudeBot |
Anthropic | |
Google-Extended |
||
CCBot |
Common Crawl | |
Amazonbot |
Amazon | |
Claude-SearchBot |
Anthropic | |
Claude-User |
Anthropic |
Google-Extended is a permission token, not a crawler: Google fetches with Googlebot, which still follows its own group. For it, a gap is a gap in the rules, not necessarily in what gets fetched.
Who blocks which AI crawler #
Share of the 5,539 robots.txt files we could parse whose rules block each crawler from the whole site — the verdict for
/. That includes files that block every crawler under User-agent: *: 238 files
(4.3%) do. The top 1,000 domains block more than the rest: 21.6% block
GPTBot, against 14.1% for ranks 1,001–10,000.
| Crawler | Used for | Blocked · all | Top 1,000 | 1,001–10,000 |
|---|---|---|---|---|
CCBot Common Crawl |
Training | 23.8% | 14.5% | |
Bytespider ByteDance |
Training | 24.8% | 14.2% | |
GPTBot OpenAI |
Training | 21.6% | 14.1% | |
ClaudeBot Anthropic |
Training | 21.4% | 13.0% | |
Google-Extended Google |
Training | 19.0% | 11.4% | |
meta-externalagent Meta |
Training | 20.0% | 11.3% | |
Applebot-Extended Apple |
Training | 18.2% | 10.9% | |
Amazonbot Amazon |
Training | 16.8% | 10.9% | |
PerplexityBot Perplexity |
Answer engine | 18.4% | 10.3% | |
ChatGPT-User OpenAI |
Answer engine | 14.9% | 9.0% | |
DuckAssistBot DuckDuckGo |
Answer engine | 14.5% | 7.7% | |
meta-externalfetcher Meta |
Other | 13.9% | 7.7% | |
Perplexity-User Perplexity |
Answer engine | 15.8% | 7.3% | |
OAI-SearchBot OpenAI |
Answer engine | 12.9% | 7.5% | |
Claude-User Anthropic |
Answer engine | 15.2% | 7.0% | |
Claude-SearchBot Anthropic |
Answer engine | 14.3% | 6.9% | |
Google-CloudVertexBot Google |
Other | 15.2% | 6.7% | |
Amzn-SearchBot Amazon |
Answer engine | 10.5% | 5.2% | |
Amzn-User Amazon |
Answer engine | 10.1% | 5.2% | |
OAI-AdsBot OpenAI |
Other | 9.1% | 4.0% |
Used for, from each vendor's documentation: training crawlers collect pages to build models; answer engines fetch pages to answer a question and can cite them; other covers ad checks and similar fetchers. Perplexity says Perplexity-User generally ignores robots.txt because a user asked for the page, so blocking it states a preference it won't enforce.
Block training, keep answers: 7.1% #
A common recommendation is to split the decision: refuse training crawlers such as GPTBot and CCBot, which give nothing back, and allow answer engines such as OAI-SearchBot and PerplexityBot, which can cite and link to you. Few sites do it. Of the 1,106 files that block at least one training crawler, 714 (64.6%) also block at least one answer engine. Only 392 (7.1% of all files) block training while leaving all 9 answer engines open.
| Pattern | Share of files | Files |
|---|---|---|
| Block at least one training crawler | 1,106 | |
| Block at least one answer engine | 727 | |
| Block training, leave every answer engine open | 392 | |
| Block all 20 | 194 |
8 training crawlers, 9 answer engines and 3 others. Blocking all 20 is almost never done by name: 193 of the 194 files that block all 20 do it by blocking every crawler.
llms.txt and Content Signals #
llms.txt, proposed in September 2024, is a Markdown file at the root of a site that gives
language models a summary and a list of key pages. Its one required element is a # title line. 878 of
the 6,082 sites that answered our request (14.4%) serve a valid one: status 200, plain
text rather than a web page, starting with a title line. The top 1,000 domains are a little ahead: 18.5%, against 14.0% for ranks 1,001–10,000.
| Response | Share of sites | Sites |
|---|---|---|
| No file (404 or 410) | 4,296 | |
| Valid llms.txt | 878 | |
| An HTML page instead of a file | 409 | |
| Redirected to a different page | 264 | |
| Plain text without a # title line | 142 | |
| Other responses | 93 |
Content Signals, published by Cloudflare in September 2025,
add a Content-Signal line to robots.txt with three yes-or-no preferences: search,
ai-input and ai-train. 150 of the 5,539 files
(2.7%) carry one, and 90 of those say
ai-train=no. The line states a preference; it is not part of RFC 9309, and crawlers are not required to read it.
WordPress sites #
We identified 396 WordPress sites from their homepage HTML. 41.3% of their robots.txt
files still contain WordPress's default rules (Disallow: /wp-admin/ and
Allow: /wp-admin/admin-ajax.php). They name AI crawlers less often than other sites and block GPTBot about as often.
| WordPress | Other sites | |
|---|---|---|
| Sites | 396 | 4,894 |
| Parseable robots.txt | 361 | 3,959 |
| Name at least one AI crawler | 16.9% | 22.0% |
| Block GPTBot from the whole site | 10.0% | 10.5% |
| Block at least one training crawler | 12.5% | 15.1% |
| Serve a valid llms.txt | 16.6% | 14.7% |
WordPress: a generator tag, an api.w.org link, or both wp-content and wp-includes paths in the homepage HTML. The 22 sites with WordPress rules in robots.txt but no trace in the homepage are left out of both columns. WordPress is uncommon this high up the list (7.5% of readable homepages), so this is a small sample.
Check your own file #
Paste your robots.txt into our free checker. It shows what Googlebot, Bingbot and 20 AI crawlers may
fetch, and lists every path a named crawler can reach that User-agent: * blocks. It runs in your browser;
nothing is uploaded.
Writing a new file? The generator on the same page copies your rules into every crawler group it adds, so none of them has a gap.
On WordPress? AI Crawler Check, our free plugin, checks your live robots.txt for these gaps from the dashboard and flags them in Site Health. It reads the file and never edits it.
FAQ #
Does User-agent: * apply to GPTBot? #
How many websites block GPTBot? #
How do I block AI training but allow AI search? #
How many sites have an llms.txt file? #
Is robots.txt enough to keep AI crawlers out? #
Method #
- Domains: the top 10,000 entries of theTranco list , a ranking of popular domains built for research (list4672X , generated 2026-10-11).
- When: one pass on 2026-10-12, 07:35–09:34 UTC.
- Crawler:HitMindBot , user agent
Mozilla/5.0 (compatible; HitMindBot/1.0; +https://hitmind.ai/bot). At most three requests per domain, one after another:/robots.txt, then/llms.txt(first 64 KB), then the homepage (first 256 KB, used only to detect WordPress). We followed each site's rules for HitMindBot, orUser-agent: *when it doesn't name us, and skipped what they disallow (270 sites for llms.txt). When robots.txt returned a server error or didn't answer, we treated the site as fully disallowed and fetched nothing else, as RFC 9309 says to. - File size: robots.txt was read up to 500 KiB, the limit Google applies. 10 files larger than 100 KB were analyzed on their first 100 KB.
- Rules: RFC 9309 group selection and longest-match precedence (Allow wins a tie), with
*and$wildcards — the same code as our publicrobots.txt checker .Blocked means the rules disallow/for that crawler.Named means the file has a group for that crawler's user agent. - AI crawlers: the 20 user agents in our robots.txt generator, each with its vendor's documentation, grouped by what the content is used for (8 training, 9 answer engines, 3 other).
- llms.txt counted as valid when it returned 200, was not an HTML page, was not redirected to a different path, and its
first non-empty line was a
#title. - Correction, October 21, 2026: this page first said 28.4% of robots.txt files name at least one AI crawler. Our
analysis also counted a crawler as named when a file had no group for it and no
User-agent: *group. The corrected share is 23.2%; the WordPress comparison and the per-crawler "named" counts in the CSV changed for the same reason. No blocking figure changed, and neither did the inheritance-gap result.
| What the 10,000 domains returned for /robots.txt | ||
|---|---|---|
| Served a robots.txt we could parse | 5,539 | 55.4% |
| --- | --- | --- |
| 403 Forbidden | 1,208 | 12.1% |
| 503 Service Unavailable | 920 | 9.2% |
| Other 5xx or 429 | 70 | 0.7% |
| No file (404 or 410) | 667 | 6.7% |
| Other 4xx | 61 | 0.6% |
| An HTML page instead of a robots.txt | 480 | 4.8% |
| No answer (DNS, TLS, timeout, refused) | 1,054 | 10.5% |
| Other | 1 | 0.0% |
Limitations #
- About a fifth of the domains refused us. 1,208 answered 403 and 920 answered 503 for robots.txt, most of them bot protection. They are left out of every share above. Sites that turn away unknown bots may turn away AI crawlers more often too, so the blocking shares are more likely too low than too high.
- Many entries are not websites. Popular-domain lists include API, CDN, ad and analytics hostnames that serve no pages, which accounts for many of the 1,054 domains that didn't answer at all.
- One snapshot from one location. A few sites serve different files by region or by user agent.
- Rules, not behavior. A gap shows what the rules permit, not what any crawler fetched, and some gaps may be deliberate. We apply RFC 9309; we can't see how each vendor's parser behaves.
- Permission tokens. Google-Extended and Applebot-Extended are not crawlers. Google and Apple fetch with Googlebot and Applebot, so a gap for either token may change nothing in practice.
- Approximations. Path types are pattern matches on the path. WordPress detection depends on the homepage, and 1,573 homepages were bot-challenge pages.
Data and citation #
Download the aggregate figures (CSV). Every number on this page comes from that file, with its denominator. We don't publish per-site results, as our crawler page promises.
While running the numbers we found and fixed four problems in our own robots.txt checker — two that reported gaps that weren't there, and two that made it hang on unusual files (one with 88 wildcards in a rule, one with 12,000 Disallow lines). Every figure here was recomputed from the stored files with the fixed code.
HitMind, “AI crawler access in the top 10,000 sites,” October 2026. https://hitmind.ai/research/ai-crawler-access-2026/
Questions about the data, or a breakdown we didn't publish: support@hitmind.ai.