{"slug": "ai-crawlers-in-robots-txt-a-study-of-the-top-10k-sites-2026", "title": "AI crawlers in robots.txt: a study of the top 10k sites (2026)", "summary": "A study of the top 10,000 websites found that 29.8% of robots.txt files that block paths for all crawlers and name an AI crawler let that crawler past at least one of those blocks (329 of 1,105 files), according to the 2026 analysis. In 182 of those files the named crawler's group contained only a blanket `Allow: /` or an empty `Disallow:`, which under RFC 9309 section 2.2.1 means the crawler stops reading the `User-agent: *` rules entirely; the paths most often let back in were site search (155 files), login and account pages (136), APIs (92), admin areas (79) and cart or checkout pages (70). The study also found 23.2% of robots.txt files name at least one AI crawler, GPTBot is blocked site-wide by 14.8% (21.6% among the top 1,000 domains), CCBot by 15.3% and Bytespider by 15.1%, while 14.4% of responding sites serve a valid llms.txt and 2.7% of robots.txt files carry Cloudflare's Content-Signal line.", "body_md": "## Key findings\n\n-  **29.8%** of the robots.txt files that block paths for everyone and name an AI crawler let that\n              crawler past at least one of those blocks (329 of 1,105). In 182 of them, the\n              crawler's group only allows everything: a blanket`Allow: /` or an empty`Disallow:` .\n- The paths most often let back in: site search (155 files), login and account pages (136), APIs (92), admin areas (79) and cart or checkout pages (70).\n- The crawlers let through most often: GPTBot (187), OAI-SearchBot (179), ChatGPT-User (178), PerplexityBot (150), ClaudeBot (139).\n-  **23.2%** of robots.txt files name at least one AI crawler. GPTBot is blocked from the whole site by**14.8%** (21.6% among the top 1,000 domains); CCBot (15.3%)\n              and Bytespider (15.1%) are blocked most often.\n-  20.0% block at least one AI training crawler, but only **7.1%** do\n              so while leaving all 9 AI answer engines open.\n-  **14.4%** of the sites that answered serve a valid llms.txt; 2.7% of robots.txt files\n              carry Cloudflare's`Content-Signal` line.\n- WordPress sites name AI crawlers less often than other sites (16.9% against 22.0%).\n\n## The rule behind it\n\nA robots.txt file is a list of groups. Each group starts with one or more `User-agent` lines and lists the rules for\n            those crawlers. The standard, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1), has a crawler\n            look for the group that names it. Only if there is none does it fall back:\n\n “If no matching group exists, crawlers MUST obey the group with a user-agent line with the \"*\" value, if present.”\n\nRFC 9309, section 2.2.1\n\nGoogle documents the same thing for its own crawlers: user agent specific groups and the global `*` group\n[are not combined](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt). So the moment a\n            file adds a group for GPTBot — even one that only says `Allow: /` to welcome it — GPTBot stops reading every rule\n            written under `User-agent: *`.\n\n```\nUser-agent: *\nDisallow: /search\nDisallow: /account/\nDisallow: /api/\n\nUser-agent: GPTBot\nAllow: /\n```\n\n| Path | Every other crawlere.g. Googlebot | GPTBothas its own group | \n|---|---|---|\n| `/search?q=…` | Blocked | Allowed | \n| `/account/settings` | Blocked | Allowed | \n| `/api/v2/orders` | Blocked | Allowed | \n| `/blog/any-post` | Allowed | Allowed | \n\nHighlighted rows: paths the file blocks for everyone except the crawler it names.\n\n### Two ways to fix it\n\n-  **If the group only exists to welcome the crawler** (it says nothing but`Allow: /` ), delete it. The crawler\n              falls back to`User-agent: *` and follows the same rules as everyone else.\n- **If the crawler needs rules of its own** , repeat every`Disallow` line from`User-agent: *` inside its group:\n\n```\nUser-agent: GPTBot\nDisallow: /search\nDisallow: /account/\nDisallow: /api/\n```\n\nOur [free robots.txt checker](https://hitmind.ai/tools/robots-txt-generator/#check) finds this for you: paste a file and it lists every path a named crawler can\n            reach that `User-agent: *` blocks.\n\nHow we count it: a file has a gap when a path is blocked under `User-agent: *` and a named AI crawler's own group has\n            no rule for that path at all, or only a blanket `Allow: /`. If the group allows a specific path on purpose (for\n            example `Allow: /api/public/`), we don't count it. Blocking the whole site under `*` while allowing named\n            crawlers is a common deliberate pattern, so `Disallow: /` on its own doesn't count either.\n\n## What gets let back in\n\nNone of these paths is secret — robots.txt is public, and it isn't access control. But they are the paths site owners block on purpose: search results that spawn endless near-duplicate URLs, account pages, and endpoints that cost money to serve.\n\n| Path type | Share of files | Files | \n|---|---|---|\n| Site search results |  | 155 | \n| Login, account and sign-up pages |  | 136 | \n| API endpoints |  | 92 | \n| Admin areas |  | 79 | \n| Cart, checkout and orders |  | 70 | \n| Feeds and XML files |  | 44 | \n\nGrouped by patterns in the blocked path (search, ?s=, login, account, /api/, admin, cart, checkout, feed and similar), so the types are approximate.\n\nThe most common shape is a group that only allows everything — a blanket `Allow: /` or an empty `Disallow:`: 182 of\n            the 329 files (55.3%). The other 147 give the crawler\n            rules of its own but leave out some of the paths blocked for everyone else.\n\n| Crawler | Files | Operator | \n|---|---|---|\n| `GPTBot` |  | OpenAI | \n| `OAI-SearchBot` |  | OpenAI | \n| `ChatGPT-User` |  | OpenAI | \n| `PerplexityBot` |  | Perplexity | \n| `ClaudeBot` |  | Anthropic | \n| `Google-Extended` |  |  | \n| `CCBot` |  | Common Crawl | \n| `Amazonbot` |  | Amazon | \n| `Claude-SearchBot` |  | Anthropic | \n| `Claude-User` |  | Anthropic | \n\nGoogle-Extended is a permission token, not a crawler: Google fetches with Googlebot, which still follows its own group. For it, a gap is a gap in the rules, not necessarily in what gets fetched.\n\n## Who blocks which AI crawler\n\nShare of the 5,539 robots.txt files we could parse whose rules block each crawler from the whole site — the verdict for\n`/`. That includes files that block every crawler under `User-agent: *`: 238 files\n            (4.3%) do. The top 1,000 domains block more than the rest: 21.6% block\n            GPTBot, against 14.1% for ranks 1,001–10,000.\n\n| Crawler | Used for | Blocked · all | Top 1,000 | 1,001–10,000 | \n|---|---|---|---|---|\n| `CCBot` Common Crawl | Training |  | 23.8% | 14.5% | \n| `Bytespider` ByteDance | Training |  | 24.8% | 14.2% | \n| `GPTBot` OpenAI | Training |  | 21.6% | 14.1% | \n| `ClaudeBot` Anthropic | Training |  | 21.4% | 13.0% | \n| `Google-Extended` Google | Training |  | 19.0% | 11.4% | \n| `meta-externalagent` Meta | Training |  | 20.0% | 11.3% | \n| `Applebot-Extended` Apple | Training |  | 18.2% | 10.9% | \n| `Amazonbot` Amazon | Training |  | 16.8% | 10.9% | \n| `PerplexityBot` Perplexity | Answer engine |  | 18.4% | 10.3% | \n| `ChatGPT-User` OpenAI | Answer engine |  | 14.9% | 9.0% | \n| `DuckAssistBot` DuckDuckGo | Answer engine |  | 14.5% | 7.7% | \n| `meta-externalfetcher` Meta | Other |  | 13.9% | 7.7% | \n| `Perplexity-User` Perplexity | Answer engine |  | 15.8% | 7.3% | \n| `OAI-SearchBot` OpenAI | Answer engine |  | 12.9% | 7.5% | \n| `Claude-User` Anthropic | Answer engine |  | 15.2% | 7.0% | \n| `Claude-SearchBot` Anthropic | Answer engine |  | 14.3% | 6.9% | \n| `Google-CloudVertexBot` Google | Other |  | 15.2% | 6.7% | \n| `Amzn-SearchBot` Amazon | Answer engine |  | 10.5% | 5.2% | \n| `Amzn-User` Amazon | Answer engine |  | 10.1% | 5.2% | \n| `OAI-AdsBot` OpenAI | Other |  | 9.1% | 4.0% | \n\nUsed for, from each vendor's documentation: training crawlers collect pages to build models; answer engines fetch pages to answer a question and can cite them; other covers ad checks and similar fetchers. Perplexity says Perplexity-User generally ignores robots.txt because a user asked for the page, so blocking it states a preference it won't enforce.\n\n## Block training, keep answers: 7.1%\n\nA common recommendation is to split the decision: refuse training crawlers such as GPTBot and CCBot, which give nothing back, and allow answer engines such as OAI-SearchBot and PerplexityBot, which can cite and link to you. Few sites do it. Of the 1,106 files that block at least one training crawler, 714 (64.6%) also block at least one answer engine. Only 392 (7.1% of all files) block training while leaving all 9 answer engines open.\n\n| Pattern | Share of files | Files | \n|---|---|---|\n| Block at least one training crawler |  | 1,106 | \n| Block at least one answer engine |  | 727 | \n| Block training, leave every answer engine open |  | 392 | \n| Block all 20 |  | 194 | \n\n8 training crawlers, 9 answer engines and 3 others. Blocking all 20 is almost never done by name: 193 of the 194 files that block all 20 do it by blocking every crawler.\n\n## llms.txt and Content Signals\n\n [llms.txt](https://llmstxt.org/), proposed in September 2024, is a Markdown file at the root of a site that gives\n            language models a summary and a list of key pages. Its one required element is a `#` title line. 878 of\n            the 6,082 sites that answered our request (14.4%) serve a valid one: status 200, plain\n            text rather than a web page, starting with a title line. The top 1,000 domains are a little ahead:  18.5%, against 14.0% for ranks 1,001–10,000.\n\n| Response | Share of sites | Sites | \n|---|---|---|\n| No file (404 or 410) |  | 4,296 | \n| Valid llms.txt |  | 878 | \n| An HTML page instead of a file |  | 409 | \n| Redirected to a different page |  | 264 | \n| Plain text without a # title line |  | 142 | \n| Other responses |  | 93 | \n\n [Content Signals](https://blog.cloudflare.com/content-signals-policy/), published by Cloudflare in September 2025,\n            add a `Content-Signal` line to robots.txt with three yes-or-no preferences: `search`,\n`ai-input` and `ai-train`. 150 of the 5,539 files\n            (2.7%) carry one, and 90 of those say\n`ai-train=no`. The line states a preference; it is not part of RFC 9309, and crawlers are not required to read it.\n\n## WordPress sites\n\nWe identified 396 WordPress sites from their homepage HTML. 41.3% of their robots.txt\n            files still contain WordPress's default rules (`Disallow: /wp-admin/` and\n`Allow: /wp-admin/admin-ajax.php`). They name AI crawlers less often than other sites and block GPTBot about as often.\n\n|  | WordPress | Other sites | \n|---|---|---|\n| Sites | 396 | 4,894 | \n| Parseable robots.txt | 361 | 3,959 | \n| Name at least one AI crawler | 16.9% | 22.0% | \n| Block GPTBot from the whole site | 10.0% | 10.5% | \n| Block at least one training crawler | 12.5% | 15.1% | \n| Serve a valid llms.txt | 16.6% | 14.7% | \n\nWordPress: a generator tag, an api.w.org link, or both wp-content and wp-includes paths in the homepage HTML. The 22 sites with WordPress rules in robots.txt but no trace in the homepage are left out of both columns. WordPress is uncommon this high up the list (7.5% of readable homepages), so this is a small sample.\n\n## Check your own file\n\nPaste your robots.txt into our free checker. It shows what Googlebot, Bingbot and 20 AI crawlers may\n              fetch, and lists every path a named crawler can reach that `User-agent: *` blocks. It runs in your browser;\n              nothing is uploaded.\n\nWriting a new file? The [generator](https://hitmind.ai/tools/robots-txt-generator/) on the same page copies your rules into every\n              crawler group it adds, so none of them has a gap.\n\nOn WordPress? [AI Crawler Check](https://wordpress.org/plugins/hitmind-ai-crawler-check/), our free plugin, checks your live robots.txt for\n              these gaps from the dashboard and flags them in Site Health. It reads the file and never edits it.\n\n## FAQ\n\n## Does User-agent: * apply to GPTBot?\n\n## How many websites block GPTBot?\n\n## How do I block AI training but allow AI search?\n\n## How many sites have an llms.txt file?\n\n## Is robots.txt enough to keep AI crawlers out?\n\n## Method\n\n-  **Domains:** the top 10,000 entries of the[Tranco list](https://tranco-list.eu/) , a ranking of\n              popular domains built for research (list[4672X](https://tranco-list.eu/list/4672X/1000000) ,\n              generated 2026-10-11).\n- **When:** one pass on 2026-10-12, 07:35–09:34 UTC.\n-  **Crawler:**[HitMindBot](https://hitmind.ai/bot) , user agent`Mozilla/5.0 (compatible; HitMindBot/1.0; +https://hitmind.ai/bot)` . At most three requests per\n              domain, one after another:`/robots.txt` , then`/llms.txt` (first 64 KB), then the homepage (first\n              256 KB, used only to detect WordPress). We followed each site's rules for HitMindBot, or`User-agent: *` when it\n              doesn't name us, and skipped what they disallow (270 sites for llms.txt). When robots.txt returned a\n              server error or didn't answer, we treated the site as fully disallowed and fetched nothing else, as RFC 9309 says to.\n-  **File size:** robots.txt was read up to 500 KiB, the limit Google applies. 10 files larger than 100 KB were analyzed on\n              their first 100 KB.\n-  **Rules:** RFC 9309 group selection and longest-match precedence (Allow wins a tie), with`*` and`$` wildcards — the same code as our public[robots.txt checker](https://hitmind.ai/tools/robots-txt-generator/#check) .**Blocked** means the\n              rules disallow`/` for that crawler.**Named** means the file has a group for that crawler's user agent.\n-  **AI crawlers:** the 20 user agents in our robots.txt generator, each with its vendor's\n              documentation, grouped by what the content is used for (8 training, 9 answer\n              engines, 3 other).\n-  **llms.txt counted as valid** when it returned 200, was not an HTML page, was not redirected to a different path, and its\n              first non-empty line was a`#` title.\n-  **Correction, October 21, 2026:** this page first said 28.4% of robots.txt files name at least one AI crawler. Our\n              analysis also counted a crawler as named when a file had no group for it and no`User-agent: *` group. The\n              corrected share is 23.2%; the WordPress comparison and the per-crawler \"named\" counts in the CSV\n              changed for the same reason. No blocking figure changed, and neither did the inheritance-gap result.\n\n| What the 10,000 domains returned for /robots.txt |  |  | \n|---|---|---|\n| Served a robots.txt we could parse | 5,539 | 55.4% | \n|---|---|---|\n| 403 Forbidden | 1,208 | 12.1% | \n| 503 Service Unavailable | 920 | 9.2% | \n| Other 5xx or 429 | 70 | 0.7% | \n| No file (404 or 410) | 667 | 6.7% | \n| Other 4xx | 61 | 0.6% | \n| An HTML page instead of a robots.txt | 480 | 4.8% | \n| No answer (DNS, TLS, timeout, refused) | 1,054 | 10.5% | \n| Other | 1 | 0.0% | \n\n## Limitations\n\n-  **About a fifth of the domains refused us.** 1,208 answered 403 and 920 answered 503 for robots.txt,\n              most of them bot protection. They are left out of every share above. Sites that turn away unknown bots may turn away AI\n              crawlers more often too, so the blocking shares are more likely too low than too high.\n-  **Many entries are not websites.** Popular-domain lists include API, CDN, ad and analytics hostnames that serve no pages,\n              which accounts for many of the 1,054 domains that didn't answer at all.\n- **One snapshot from one location.** A few sites serve different files by region or by user agent.\n-  **Rules, not behavior.** A gap shows what the rules permit, not what any crawler fetched, and some gaps may be deliberate.\n              We apply RFC 9309; we can't see how each vendor's parser behaves.\n-  **Permission tokens.** Google-Extended and Applebot-Extended are not crawlers. Google and Apple fetch with Googlebot and\n              Applebot, so a gap for either token may change nothing in practice.\n-  **Approximations.** Path types are pattern matches on the path. WordPress detection depends on the homepage, and  1,573 homepages were bot-challenge pages.\n\n## Data and citation\n\n [Download the aggregate figures (CSV)](https://hitmind.ai/research/ai-crawler-access-2026/hitmind-ai-crawler-access-2026.csv). Every number on this page comes from that file, with its\n            denominator. We don't publish per-site results, [as our crawler page promises](https://hitmind.ai/bot#research).\n\nWhile running the numbers we found and fixed four problems in our own robots.txt checker — two that reported gaps that weren't there, and two that made it hang on unusual files (one with 88 wildcards in a rule, one with 12,000 Disallow lines). Every figure here was recomputed from the stored files with the fixed code.\n\nHitMind, “AI crawler access in the top 10,000 sites,” October 2026. https://hitmind.ai/research/ai-crawler-access-2026/\n\nQuestions about the data, or a breakdown we didn't publish: [support@hitmind.ai](mailto:support@hitmind.ai).", "url": "https://wpnews.pro/news/ai-crawlers-in-robots-txt-a-study-of-the-top-10k-sites-2026", "canonical_source": "https://hitmind.ai/research/ai-crawler-access-2026/", "published_at": "2026-10-28 12:57:54+00:00", "updated_at": "2026-10-28 13:12:48.298751+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "agent-protocols"], "entities": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "PerplexityBot", "ClaudeBot", "CCBot", "Bytespider", "Cloudflare"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-crawlers-in-robots-txt-a-study-of-the-top-10k-sites-2026", "markdown": "https://wpnews.pro/news/ai-crawlers-in-robots-txt-a-study-of-the-top-10k-sites-2026.md", "text": "https://wpnews.pro/news/ai-crawlers-in-robots-txt-a-study-of-the-top-10k-sites-2026.txt", "jsonld": "https://wpnews.pro/news/ai-crawlers-in-robots-txt-a-study-of-the-top-10k-sites-2026.jsonld"}}