# AI crawlers in robots.txt: a study of the top 10k sites (2026)

> Source: <https://hitmind.ai/research/ai-crawler-access-2026/>
> Published: 2026-10-28 12:57:54+00:00

## Key findings

-  **29.8%** of the robots.txt files that block paths for everyone and name an AI crawler let that
              crawler past at least one of those blocks (329 of 1,105). In 182 of them, the
              crawler's group only allows everything: a blanket`Allow: /` or an empty`Disallow:` .
- The paths most often let back in: site search (155 files), login and account pages (136), APIs (92), admin areas (79) and cart or checkout pages (70).
- The crawlers let through most often: GPTBot (187), OAI-SearchBot (179), ChatGPT-User (178), PerplexityBot (150), ClaudeBot (139).
-  **23.2%** of robots.txt files name at least one AI crawler. GPTBot is blocked from the whole site by**14.8%** (21.6% among the top 1,000 domains); CCBot (15.3%)
              and Bytespider (15.1%) are blocked most often.
-  20.0% block at least one AI training crawler, but only **7.1%** do
              so while leaving all 9 AI answer engines open.
-  **14.4%** of the sites that answered serve a valid llms.txt; 2.7% of robots.txt files
              carry Cloudflare's`Content-Signal` line.
- WordPress sites name AI crawlers less often than other sites (16.9% against 22.0%).

## The rule behind it

A robots.txt file is a list of groups. Each group starts with one or more `User-agent` lines and lists the rules for
            those crawlers. The standard, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1), has a crawler
            look for the group that names it. Only if there is none does it fall back:

 “If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.”

RFC 9309, section 2.2.1

Google documents the same thing for its own crawlers: user agent specific groups and the global `*` group
[are not combined](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt). So the moment a
            file adds a group for GPTBot — even one that only says `Allow: /` to welcome it — GPTBot stops reading every rule
            written under `User-agent: *`.

```
User-agent: *
Disallow: /search
Disallow: /account/
Disallow: /api/

User-agent: GPTBot
Allow: /
```

| Path | Every other crawlere.g. Googlebot | GPTBothas its own group | 
|---|---|---|
| `/search?q=…` | Blocked | Allowed | 
| `/account/settings` | Blocked | Allowed | 
| `/api/v2/orders` | Blocked | Allowed | 
| `/blog/any-post` | Allowed | Allowed | 

Highlighted rows: paths the file blocks for everyone except the crawler it names.

### Two ways to fix it

-  **If the group only exists to welcome the crawler** (it says nothing but`Allow: /` ), delete it. The crawler
              falls back to`User-agent: *` and follows the same rules as everyone else.
- **If the crawler needs rules of its own** , repeat every`Disallow` line from`User-agent: *` inside its group:

```
User-agent: GPTBot
Disallow: /search
Disallow: /account/
Disallow: /api/
```

Our [free robots.txt checker](https://hitmind.ai/tools/robots-txt-generator/#check) finds this for you: paste a file and it lists every path a named crawler can
            reach that `User-agent: *` blocks.

How we count it: a file has a gap when a path is blocked under `User-agent: *` and a named AI crawler's own group has
            no rule for that path at all, or only a blanket `Allow: /`. If the group allows a specific path on purpose (for
            example `Allow: /api/public/`), we don't count it. Blocking the whole site under `*` while allowing named
            crawlers is a common deliberate pattern, so `Disallow: /` on its own doesn't count either.

## What gets let back in

None of these paths is secret — robots.txt is public, and it isn't access control. But they are the paths site owners block on purpose: search results that spawn endless near-duplicate URLs, account pages, and endpoints that cost money to serve.

| Path type | Share of files | Files | 
|---|---|---|
| Site search results |  | 155 | 
| Login, account and sign-up pages |  | 136 | 
| API endpoints |  | 92 | 
| Admin areas |  | 79 | 
| Cart, checkout and orders |  | 70 | 
| Feeds and XML files |  | 44 | 

Grouped by patterns in the blocked path (search, ?s=, login, account, /api/, admin, cart, checkout, feed and similar), so the types are approximate.

The most common shape is a group that only allows everything — a blanket `Allow: /` or an empty `Disallow:`: 182 of
            the 329 files (55.3%). The other 147 give the crawler
            rules of its own but leave out some of the paths blocked for everyone else.

| Crawler | Files | Operator | 
|---|---|---|
| `GPTBot` |  | OpenAI | 
| `OAI-SearchBot` |  | OpenAI | 
| `ChatGPT-User` |  | OpenAI | 
| `PerplexityBot` |  | Perplexity | 
| `ClaudeBot` |  | Anthropic | 
| `Google-Extended` |  |  | 
| `CCBot` |  | Common Crawl | 
| `Amazonbot` |  | Amazon | 
| `Claude-SearchBot` |  | Anthropic | 
| `Claude-User` |  | Anthropic | 

Google-Extended is a permission token, not a crawler: Google fetches with Googlebot, which still follows its own group. For it, a gap is a gap in the rules, not necessarily in what gets fetched.

## Who blocks which AI crawler

Share of the 5,539 robots.txt files we could parse whose rules block each crawler from the whole site — the verdict for
`/`. That includes files that block every crawler under `User-agent: *`: 238 files
            (4.3%) do. The top 1,000 domains block more than the rest: 21.6% block
            GPTBot, against 14.1% for ranks 1,001–10,000.

| Crawler | Used for | Blocked · all | Top 1,000 | 1,001–10,000 | 
|---|---|---|---|---|
| `CCBot` Common Crawl | Training |  | 23.8% | 14.5% | 
| `Bytespider` ByteDance | Training |  | 24.8% | 14.2% | 
| `GPTBot` OpenAI | Training |  | 21.6% | 14.1% | 
| `ClaudeBot` Anthropic | Training |  | 21.4% | 13.0% | 
| `Google-Extended` Google | Training |  | 19.0% | 11.4% | 
| `meta-externalagent` Meta | Training |  | 20.0% | 11.3% | 
| `Applebot-Extended` Apple | Training |  | 18.2% | 10.9% | 
| `Amazonbot` Amazon | Training |  | 16.8% | 10.9% | 
| `PerplexityBot` Perplexity | Answer engine |  | 18.4% | 10.3% | 
| `ChatGPT-User` OpenAI | Answer engine |  | 14.9% | 9.0% | 
| `DuckAssistBot` DuckDuckGo | Answer engine |  | 14.5% | 7.7% | 
| `meta-externalfetcher` Meta | Other |  | 13.9% | 7.7% | 
| `Perplexity-User` Perplexity | Answer engine |  | 15.8% | 7.3% | 
| `OAI-SearchBot` OpenAI | Answer engine |  | 12.9% | 7.5% | 
| `Claude-User` Anthropic | Answer engine |  | 15.2% | 7.0% | 
| `Claude-SearchBot` Anthropic | Answer engine |  | 14.3% | 6.9% | 
| `Google-CloudVertexBot` Google | Other |  | 15.2% | 6.7% | 
| `Amzn-SearchBot` Amazon | Answer engine |  | 10.5% | 5.2% | 
| `Amzn-User` Amazon | Answer engine |  | 10.1% | 5.2% | 
| `OAI-AdsBot` OpenAI | Other |  | 9.1% | 4.0% | 

Used for, from each vendor's documentation: training crawlers collect pages to build models; answer engines fetch pages to answer a question and can cite them; other covers ad checks and similar fetchers. Perplexity says Perplexity-User generally ignores robots.txt because a user asked for the page, so blocking it states a preference it won't enforce.

## Block training, keep answers: 7.1%

A common recommendation is to split the decision: refuse training crawlers such as GPTBot and CCBot, which give nothing back, and allow answer engines such as OAI-SearchBot and PerplexityBot, which can cite and link to you. Few sites do it. Of the 1,106 files that block at least one training crawler, 714 (64.6%) also block at least one answer engine. Only 392 (7.1% of all files) block training while leaving all 9 answer engines open.

| Pattern | Share of files | Files | 
|---|---|---|
| Block at least one training crawler |  | 1,106 | 
| Block at least one answer engine |  | 727 | 
| Block training, leave every answer engine open |  | 392 | 
| Block all 20 |  | 194 | 

8 training crawlers, 9 answer engines and 3 others. Blocking all 20 is almost never done by name: 193 of the 194 files that block all 20 do it by blocking every crawler.

## llms.txt and Content Signals

 [llms.txt](https://llmstxt.org/), proposed in September 2024, is a Markdown file at the root of a site that gives
            language models a summary and a list of key pages. Its one required element is a `#` title line. 878 of
            the 6,082 sites that answered our request (14.4%) serve a valid one: status 200, plain
            text rather than a web page, starting with a title line. The top 1,000 domains are a little ahead:  18.5%, against 14.0% for ranks 1,001–10,000.

| Response | Share of sites | Sites | 
|---|---|---|
| No file (404 or 410) |  | 4,296 | 
| Valid llms.txt |  | 878 | 
| An HTML page instead of a file |  | 409 | 
| Redirected to a different page |  | 264 | 
| Plain text without a # title line |  | 142 | 
| Other responses |  | 93 | 

 [Content Signals](https://blog.cloudflare.com/content-signals-policy/), published by Cloudflare in September 2025,
            add a `Content-Signal` line to robots.txt with three yes-or-no preferences: `search`,
`ai-input` and `ai-train`. 150 of the 5,539 files
            (2.7%) carry one, and 90 of those say
`ai-train=no`. The line states a preference; it is not part of RFC 9309, and crawlers are not required to read it.

## WordPress sites

We identified 396 WordPress sites from their homepage HTML. 41.3% of their robots.txt
            files still contain WordPress's default rules (`Disallow: /wp-admin/` and
`Allow: /wp-admin/admin-ajax.php`). They name AI crawlers less often than other sites and block GPTBot about as often.

|  | WordPress | Other sites | 
|---|---|---|
| Sites | 396 | 4,894 | 
| Parseable robots.txt | 361 | 3,959 | 
| Name at least one AI crawler | 16.9% | 22.0% | 
| Block GPTBot from the whole site | 10.0% | 10.5% | 
| Block at least one training crawler | 12.5% | 15.1% | 
| Serve a valid llms.txt | 16.6% | 14.7% | 

WordPress: a generator tag, an api.w.org link, or both wp-content and wp-includes paths in the homepage HTML. The 22 sites with WordPress rules in robots.txt but no trace in the homepage are left out of both columns. WordPress is uncommon this high up the list (7.5% of readable homepages), so this is a small sample.

## Check your own file

Paste your robots.txt into our free checker. It shows what Googlebot, Bingbot and 20 AI crawlers may
              fetch, and lists every path a named crawler can reach that `User-agent: *` blocks. It runs in your browser;
              nothing is uploaded.

Writing a new file? The [generator](https://hitmind.ai/tools/robots-txt-generator/) on the same page copies your rules into every
              crawler group it adds, so none of them has a gap.

On WordPress? [AI Crawler Check](https://wordpress.org/plugins/hitmind-ai-crawler-check/), our free plugin, checks your live robots.txt for
              these gaps from the dashboard and flags them in Site Health. It reads the file and never edits it.

## FAQ

## Does User-agent: * apply to GPTBot?

## How many websites block GPTBot?

## How do I block AI training but allow AI search?

## How many sites have an llms.txt file?

## Is robots.txt enough to keep AI crawlers out?

## Method

-  **Domains:** the top 10,000 entries of the[Tranco list](https://tranco-list.eu/) , a ranking of
              popular domains built for research (list[4672X](https://tranco-list.eu/list/4672X/1000000) ,
              generated 2026-10-11).
- **When:** one pass on 2026-10-12, 07:35–09:34 UTC.
-  **Crawler:**[HitMindBot](https://hitmind.ai/bot) , user agent`Mozilla/5.0 (compatible; HitMindBot/1.0; +https://hitmind.ai/bot)` . At most three requests per
              domain, one after another:`/robots.txt` , then`/llms.txt` (first 64 KB), then the homepage (first
              256 KB, used only to detect WordPress). We followed each site's rules for HitMindBot, or`User-agent: *` when it
              doesn't name us, and skipped what they disallow (270 sites for llms.txt). When robots.txt returned a
              server error or didn't answer, we treated the site as fully disallowed and fetched nothing else, as RFC 9309 says to.
-  **File size:** robots.txt was read up to 500 KiB, the limit Google applies. 10 files larger than 100 KB were analyzed on
              their first 100 KB.
-  **Rules:** RFC 9309 group selection and longest-match precedence (Allow wins a tie), with`*` and`$` wildcards — the same code as our public[robots.txt checker](https://hitmind.ai/tools/robots-txt-generator/#check) .**Blocked** means the
              rules disallow`/` for that crawler.**Named** means the file has a group for that crawler's user agent.
-  **AI crawlers:** the 20 user agents in our robots.txt generator, each with its vendor's
              documentation, grouped by what the content is used for (8 training, 9 answer
              engines, 3 other).
-  **llms.txt counted as valid** when it returned 200, was not an HTML page, was not redirected to a different path, and its
              first non-empty line was a`#` title.
-  **Correction, October 21, 2026:** this page first said 28.4% of robots.txt files name at least one AI crawler. Our
              analysis also counted a crawler as named when a file had no group for it and no`User-agent: *` group. The
              corrected share is 23.2%; the WordPress comparison and the per-crawler "named" counts in the CSV
              changed for the same reason. No blocking figure changed, and neither did the inheritance-gap result.

| What the 10,000 domains returned for /robots.txt |  |  | 
|---|---|---|
| Served a robots.txt we could parse | 5,539 | 55.4% | 
|---|---|---|
| 403 Forbidden | 1,208 | 12.1% | 
| 503 Service Unavailable | 920 | 9.2% | 
| Other 5xx or 429 | 70 | 0.7% | 
| No file (404 or 410) | 667 | 6.7% | 
| Other 4xx | 61 | 0.6% | 
| An HTML page instead of a robots.txt | 480 | 4.8% | 
| No answer (DNS, TLS, timeout, refused) | 1,054 | 10.5% | 
| Other | 1 | 0.0% | 

## Limitations

-  **About a fifth of the domains refused us.** 1,208 answered 403 and 920 answered 503 for robots.txt,
              most of them bot protection. They are left out of every share above. Sites that turn away unknown bots may turn away AI
              crawlers more often too, so the blocking shares are more likely too low than too high.
-  **Many entries are not websites.** Popular-domain lists include API, CDN, ad and analytics hostnames that serve no pages,
              which accounts for many of the 1,054 domains that didn't answer at all.
- **One snapshot from one location.** A few sites serve different files by region or by user agent.
-  **Rules, not behavior.** A gap shows what the rules permit, not what any crawler fetched, and some gaps may be deliberate.
              We apply RFC 9309; we can't see how each vendor's parser behaves.
-  **Permission tokens.** Google-Extended and Applebot-Extended are not crawlers. Google and Apple fetch with Googlebot and
              Applebot, so a gap for either token may change nothing in practice.
-  **Approximations.** Path types are pattern matches on the path. WordPress detection depends on the homepage, and  1,573 homepages were bot-challenge pages.

## Data and citation

 [Download the aggregate figures (CSV)](https://hitmind.ai/research/ai-crawler-access-2026/hitmind-ai-crawler-access-2026.csv). Every number on this page comes from that file, with its
            denominator. We don't publish per-site results, [as our crawler page promises](https://hitmind.ai/bot#research).

While running the numbers we found and fixed four problems in our own robots.txt checker — two that reported gaps that weren't there, and two that made it hang on unusual files (one with 88 wildcards in a rule, one with 12,000 Disallow lines). Every figure here was recomputed from the stored files with the fixed code.

HitMind, “AI crawler access in the top 10,000 sites,” October 2026. https://hitmind.ai/research/ai-crawler-access-2026/

Questions about the data, or a breakdown we didn't publish: [support@hitmind.ai](mailto:support@hitmind.ai).