AI crawlers in robots.txt: a study of the top 10k sites (2026)

A study of the top 10,000 websites found that 29.8% of robots.txt files that block paths for all crawlers and name an AI crawler let that crawler past at least one of those blocks (329 of 1,105 files), according to the 2026 analysis. In 182 of those files the named crawler's group contained only a blanket `Allow: /` or an empty `Disallow:`, which under RFC 9309 section 2.2.1 means the crawler stops reading the `User-agent: *` rules entirely; the paths most often let back in were site search (155 files), login and account pages (136), APIs (92), admin areas (79) and cart or checkout pages (70). The study also found 23.2% of robots.txt files name at least one AI crawler, GPTBot is blocked site-wide by 14.8% (21.6% among the top 1,000 domains), CCBot by 15.3% and Bytespider by 15.1%, while 14.4% of responding sites serve a valid llms.txt and 2.7% of robots.txt files carry Cloudflare's Content-Signal line.

Key findings - 29.8% of the robots.txt files that block paths for everyone and name an AI crawler let that crawler past at least one of those blocks 329 of 1,105 . In 182 of them, the crawler's group only allows everything: a blanket Allow: / or an empty Disallow: . - The paths most often let back in: site search 155 files , login and account pages 136 , APIs 92 , admin areas 79 and cart or checkout pages 70 . - The crawlers let through most often: GPTBot 187 , OAI-SearchBot 179 , ChatGPT-User 178 , PerplexityBot 150 , ClaudeBot 139 . - 23.2% of robots.txt files name at least one AI crawler. GPTBot is blocked from the whole site by 14.8% 21.6% among the top 1,000 domains ; CCBot 15.3% and Bytespider 15.1% are blocked most often. - 20.0% block at least one AI training crawler, but only 7.1% do so while leaving all 9 AI answer engines open. - 14.4% of the sites that answered serve a valid llms.txt; 2.7% of robots.txt files carry Cloudflare's Content-Signal line. - WordPress sites name AI crawlers less often than other sites 16.9% against 22.0% . The rule behind it A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and lists the rules for those crawlers. The standard, RFC 9309 https://www.rfc-editor.org/rfc/rfc9309.html section-2.2.1 , has a crawler look for the group that names it. Only if there is none does it fall back: “If no matching group exists, crawlers MUST obey the group with a user-agent line with the " " value, if present.” RFC 9309, section 2.2.1 Google documents the same thing for its own crawlers: user agent specific groups and the global group are not combined https://developers.google.com/search/docs/crawling-indexing/robots/robots txt . So the moment a file adds a group for GPTBot — even one that only says Allow: / to welcome it — GPTBot stops reading every rule written under User-agent: . User-agent: Disallow: /search Disallow: /account/ Disallow: /api/ User-agent: GPTBot Allow: / | Path | Every other crawlere.g. Googlebot | GPTBothas its own group | |---|---|---| | /search?q=… | Blocked | Allowed | | /account/settings | Blocked | Allowed | | /api/v2/orders | Blocked | Allowed | | /blog/any-post | Allowed | Allowed | Highlighted rows: paths the file blocks for everyone except the crawler it names. Two ways to fix it - If the group only exists to welcome the crawler it says nothing but Allow: / , delete it. The crawler falls back to User-agent: and follows the same rules as everyone else. - If the crawler needs rules of its own , repeat every Disallow line from User-agent: inside its group: User-agent: GPTBot Disallow: /search Disallow: /account/ Disallow: /api/ Our free robots.txt checker https://hitmind.ai/tools/robots-txt-generator/ check finds this for you: paste a file and it lists every path a named crawler can reach that User-agent: blocks. How we count it: a file has a gap when a path is blocked under User-agent: and a named AI crawler's own group has no rule for that path at all, or only a blanket Allow: / . If the group allows a specific path on purpose for example Allow: /api/public/ , we don't count it. Blocking the whole site under while allowing named crawlers is a common deliberate pattern, so Disallow: / on its own doesn't count either. What gets let back in None of these paths is secret — robots.txt is public, and it isn't access control. But they are the paths site owners block on purpose: search results that spawn endless near-duplicate URLs, account pages, and endpoints that cost money to serve. | Path type | Share of files | Files | |---|---|---| | Site search results | | 155 | | Login, account and sign-up pages | | 136 | | API endpoints | | 92 | | Admin areas | | 79 | | Cart, checkout and orders | | 70 | | Feeds and XML files | | 44 | Grouped by patterns in the blocked path search, ?s=, login, account, /api/, admin, cart, checkout, feed and similar , so the types are approximate. The most common shape is a group that only allows everything — a blanket Allow: / or an empty Disallow: : 182 of the 329 files 55.3% . The other 147 give the crawler rules of its own but leave out some of the paths blocked for everyone else. | Crawler | Files | Operator | |---|---|---| | GPTBot | | OpenAI | | OAI-SearchBot | | OpenAI | | ChatGPT-User | | OpenAI | | PerplexityBot | | Perplexity | | ClaudeBot | | Anthropic | | Google-Extended | | | | CCBot | | Common Crawl | | Amazonbot | | Amazon | | Claude-SearchBot | | Anthropic | | Claude-User | | Anthropic | Google-Extended is a permission token, not a crawler: Google fetches with Googlebot, which still follows its own group. For it, a gap is a gap in the rules, not necessarily in what gets fetched. Who blocks which AI crawler Share of the 5,539 robots.txt files we could parse whose rules block each crawler from the whole site — the verdict for / . That includes files that block every crawler under User-agent: : 238 files 4.3% do. The top 1,000 domains block more than the rest: 21.6% block GPTBot, against 14.1% for ranks 1,001–10,000. | Crawler | Used for | Blocked · all | Top 1,000 | 1,001–10,000 | |---|---|---|---|---| | CCBot Common Crawl | Training | | 23.8% | 14.5% | | Bytespider ByteDance | Training | | 24.8% | 14.2% | | GPTBot OpenAI | Training | | 21.6% | 14.1% | | ClaudeBot Anthropic | Training | | 21.4% | 13.0% | | Google-Extended Google | Training | | 19.0% | 11.4% | | meta-externalagent Meta | Training | | 20.0% | 11.3% | | Applebot-Extended Apple | Training | | 18.2% | 10.9% | | Amazonbot Amazon | Training | | 16.8% | 10.9% | | PerplexityBot Perplexity | Answer engine | | 18.4% | 10.3% | | ChatGPT-User OpenAI | Answer engine | | 14.9% | 9.0% | | DuckAssistBot DuckDuckGo | Answer engine | | 14.5% | 7.7% | | meta-externalfetcher Meta | Other | | 13.9% | 7.7% | | Perplexity-User Perplexity | Answer engine | | 15.8% | 7.3% | | OAI-SearchBot OpenAI | Answer engine | | 12.9% | 7.5% | | Claude-User Anthropic | Answer engine | | 15.2% | 7.0% | | Claude-SearchBot Anthropic | Answer engine | | 14.3% | 6.9% | | Google-CloudVertexBot Google | Other | | 15.2% | 6.7% | | Amzn-SearchBot Amazon | Answer engine | | 10.5% | 5.2% | | Amzn-User Amazon | Answer engine | | 10.1% | 5.2% | | OAI-AdsBot OpenAI | Other | | 9.1% | 4.0% | Used for, from each vendor's documentation: training crawlers collect pages to build models; answer engines fetch pages to answer a question and can cite them; other covers ad checks and similar fetchers. Perplexity says Perplexity-User generally ignores robots.txt because a user asked for the page, so blocking it states a preference it won't enforce. Block training, keep answers: 7.1% A common recommendation is to split the decision: refuse training crawlers such as GPTBot and CCBot, which give nothing back, and allow answer engines such as OAI-SearchBot and PerplexityBot, which can cite and link to you. Few sites do it. Of the 1,106 files that block at least one training crawler, 714 64.6% also block at least one answer engine. Only 392 7.1% of all files block training while leaving all 9 answer engines open. | Pattern | Share of files | Files | |---|---|---| | Block at least one training crawler | | 1,106 | | Block at least one answer engine | | 727 | | Block training, leave every answer engine open | | 392 | | Block all 20 | | 194 | 8 training crawlers, 9 answer engines and 3 others. Blocking all 20 is almost never done by name: 193 of the 194 files that block all 20 do it by blocking every crawler. llms.txt and Content Signals llms.txt https://llmstxt.org/ , proposed in September 2024, is a Markdown file at the root of a site that gives language models a summary and a list of key pages. Its one required element is a title line. 878 of the 6,082 sites that answered our request 14.4% serve a valid one: status 200, plain text rather than a web page, starting with a title line. The top 1,000 domains are a little ahead: 18.5%, against 14.0% for ranks 1,001–10,000. | Response | Share of sites | Sites | |---|---|---| | No file 404 or 410 | | 4,296 | | Valid llms.txt | | 878 | | An HTML page instead of a file | | 409 | | Redirected to a different page | | 264 | | Plain text without a title line | | 142 | | Other responses | | 93 | Content Signals https://blog.cloudflare.com/content-signals-policy/ , published by Cloudflare in September 2025, add a Content-Signal line to robots.txt with three yes-or-no preferences: search , ai-input and ai-train . 150 of the 5,539 files 2.7% carry one, and 90 of those say ai-train=no . The line states a preference; it is not part of RFC 9309, and crawlers are not required to read it. WordPress sites We identified 396 WordPress sites from their homepage HTML. 41.3% of their robots.txt files still contain WordPress's default rules Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php . They name AI crawlers less often than other sites and block GPTBot about as often. | | WordPress | Other sites | |---|---|---| | Sites | 396 | 4,894 | | Parseable robots.txt | 361 | 3,959 | | Name at least one AI crawler | 16.9% | 22.0% | | Block GPTBot from the whole site | 10.0% | 10.5% | | Block at least one training crawler | 12.5% | 15.1% | | Serve a valid llms.txt | 16.6% | 14.7% | WordPress: a generator tag, an api.w.org link, or both wp-content and wp-includes paths in the homepage HTML. The 22 sites with WordPress rules in robots.txt but no trace in the homepage are left out of both columns. WordPress is uncommon this high up the list 7.5% of readable homepages , so this is a small sample. Check your own file Paste your robots.txt into our free checker. It shows what Googlebot, Bingbot and 20 AI crawlers may fetch, and lists every path a named crawler can reach that User-agent: blocks. It runs in your browser; nothing is uploaded. Writing a new file? The generator https://hitmind.ai/tools/robots-txt-generator/ on the same page copies your rules into every crawler group it adds, so none of them has a gap. On WordPress? AI Crawler Check https://wordpress.org/plugins/hitmind-ai-crawler-check/ , our free plugin, checks your live robots.txt for these gaps from the dashboard and flags them in Site Health. It reads the file and never edits it. FAQ Does User-agent: apply to GPTBot? How many websites block GPTBot? How do I block AI training but allow AI search? How many sites have an llms.txt file? Is robots.txt enough to keep AI crawlers out? Method - Domains: the top 10,000 entries of the Tranco list https://tranco-list.eu/ , a ranking of popular domains built for research list 4672X https://tranco-list.eu/list/4672X/1000000 , generated 2026-10-11 . - When: one pass on 2026-10-12, 07:35–09:34 UTC. - Crawler: HitMindBot https://hitmind.ai/bot , user agent Mozilla/5.0 compatible; HitMindBot/1.0; +https://hitmind.ai/bot . At most three requests per domain, one after another: /robots.txt , then /llms.txt first 64 KB , then the homepage first 256 KB, used only to detect WordPress . We followed each site's rules for HitMindBot, or User-agent: when it doesn't name us, and skipped what they disallow 270 sites for llms.txt . When robots.txt returned a server error or didn't answer, we treated the site as fully disallowed and fetched nothing else, as RFC 9309 says to. - File size: robots.txt was read up to 500 KiB, the limit Google applies. 10 files larger than 100 KB were analyzed on their first 100 KB. - Rules: RFC 9309 group selection and longest-match precedence Allow wins a tie , with and $ wildcards — the same code as our public robots.txt checker https://hitmind.ai/tools/robots-txt-generator/ check . Blocked means the rules disallow / for that crawler. Named means the file has a group for that crawler's user agent. - AI crawlers: the 20 user agents in our robots.txt generator, each with its vendor's documentation, grouped by what the content is used for 8 training, 9 answer engines, 3 other . - llms.txt counted as valid when it returned 200, was not an HTML page, was not redirected to a different path, and its first non-empty line was a title. - Correction, October 21, 2026: this page first said 28.4% of robots.txt files name at least one AI crawler. Our analysis also counted a crawler as named when a file had no group for it and no User-agent: group. The corrected share is 23.2%; the WordPress comparison and the per-crawler "named" counts in the CSV changed for the same reason. No blocking figure changed, and neither did the inheritance-gap result. | What the 10,000 domains returned for /robots.txt | | | |---|---|---| | Served a robots.txt we could parse | 5,539 | 55.4% | |---|---|---| | 403 Forbidden | 1,208 | 12.1% | | 503 Service Unavailable | 920 | 9.2% | | Other 5xx or 429 | 70 | 0.7% | | No file 404 or 410 | 667 | 6.7% | | Other 4xx | 61 | 0.6% | | An HTML page instead of a robots.txt | 480 | 4.8% | | No answer DNS, TLS, timeout, refused | 1,054 | 10.5% | | Other | 1 | 0.0% | Limitations - About a fifth of the domains refused us. 1,208 answered 403 and 920 answered 503 for robots.txt, most of them bot protection. They are left out of every share above. Sites that turn away unknown bots may turn away AI crawlers more often too, so the blocking shares are more likely too low than too high. - Many entries are not websites. Popular-domain lists include API, CDN, ad and analytics hostnames that serve no pages, which accounts for many of the 1,054 domains that didn't answer at all. - One snapshot from one location. A few sites serve different files by region or by user agent. - Rules, not behavior. A gap shows what the rules permit, not what any crawler fetched, and some gaps may be deliberate. We apply RFC 9309; we can't see how each vendor's parser behaves. - Permission tokens. Google-Extended and Applebot-Extended are not crawlers. Google and Apple fetch with Googlebot and Applebot, so a gap for either token may change nothing in practice. - Approximations. Path types are pattern matches on the path. WordPress detection depends on the homepage, and 1,573 homepages were bot-challenge pages. Data and citation Download the aggregate figures CSV https://hitmind.ai/research/ai-crawler-access-2026/hitmind-ai-crawler-access-2026.csv . Every number on this page comes from that file, with its denominator. We don't publish per-site results, as our crawler page promises https://hitmind.ai/bot research . While running the numbers we found and fixed four problems in our own robots.txt checker — two that reported gaps that weren't there, and two that made it hang on unusual files one with 88 wildcards in a rule, one with 12,000 Disallow lines . Every figure here was recomputed from the stored files with the fixed code. HitMind, “AI crawler access in the top 10,000 sites,” October 2026. https://hitmind.ai/research/ai-crawler-access-2026/ Questions about the data, or a breakdown we didn't publish: support@hitmind.ai mailto:support@hitmind.ai .