AI Access Index, Nov 2026: how 684 of the topk hostnames treat AI crawlers The first AI Access Index, published November 1, 2026, found that 79.7% (545) of the 684 top-1,000 hostnames that answered are open to AI crawlers for both search and training, while 7.7% (53) block AI entirely and 8.8% (60) allow AI search but not training. The index also measured server behavior against robots.txt, finding that 75% of the 80 sites blocking GPTBot in robots.txt still served its requests, and Bytespider (Bytedance) was the most-blocked crawler at 17.5% (120 sites), followed by CCBot at 17.3% (118) and GPTBot at 15.6% (107). How the top 1,000 websites treat AI crawlers On the 1st of every month, the same 1,000 sites, checked the same way: what each site's robots.txt tells AI crawlers, and what its server actually does when they arrive. It is not a ranking. Open or closed, a policy is a choice; the index measures which choices sites make, and whether their servers keep to them. What robots.txt tells AI crawlers Each site's policy is read the way its own report reads it: from what its robots.txt says to the 40 crawlers that fetch pages for AI search and answers, and the 9 that gather training data. - Open to AI79.7% 545 AI search: yes · AI training: yes - AI search, no AI training8.8% 60 AI search: yes · AI training: no - Selective3.8% 26 Some AI companies in, others kept out - Closed to AI7.7% 53 AI search: no · AI training: no Shares of the 684 sites that answered. Open includes 119 sites with no robots.txt at all, which allows everything. Closed includes 13 hosts that close themselves to every crawler, search engines included: mostly API and CDN hostnames rather than a decision about AI. Does the server do what robots.txt says? robots.txt is a request, and the server answers first. After the check's own visit, each homepage was asked for again four times, each time with one AI crawler's user-agent string, and the answer was set beside what the site's robots.txt says about that crawler. | Crawler | Blocked in robots.txt | Refused by the server | Refused while robots.txt allows it | Served while robots.txt blocks it | |---|---|---|---|---| | GPTBot OpenAI training | 15.6% | 8.8% | 5.9% | 75% of 80 | | OAI-SearchBot ChatGPT search | 8.2% | 3.8% | 2.3% | 76.1% of 46 | | ClaudeBot Anthropic training | 15.5% | 8.7% | 5.5% | 74.7% of 75 | | PerplexityBot Perplexity search | 12.3% | 8.5% | 6.3% | 71.7% of 60 | Blocked in robots.txt: share of the 684 sites that answered. The server columns: share of the sites whose homepage opened for the check's own request about 525 . Served while blocked: share of those that block that crawler. A refusal is a 401, 403, 429 or 503. What changed since last month This is the first edition. From the December 2026 edition on, this section names every site that changed its AI policy, started or stopped blocking one of the four crawlers, or whose server started or stopped refusing a crawler its robots.txt allows. Look up a site Every site in the index, as it answered on November 1, 2026. A site that is not in the sample can be checked the same way, free. The most-blocked AI crawlers | Crawler | Company | What it does | Blocked by | |---|---|---|---| | Bytespider https://www.crawlreadiness.com/crawlers/bytespider | Bytedance | Training only | 17.5% 120 | | CCBot https://www.crawlreadiness.com/crawlers/ccbot | Common Crawl | Training only | 17.3% 118 | | GPTBot https://www.crawlreadiness.com/crawlers/gptbot | OpenAI | Training only | 15.6% 107 | | ClaudeBot https://www.crawlreadiness.com/crawlers/claudebot | Anthropic | Training only | 15.5% 106 | | Diffbot https://www.crawlreadiness.com/crawlers/diffbot | Diffbot | AI answers | 14% 96 | | meta-externalagent https://www.crawlreadiness.com/crawlers/meta-externalagent | Meta AI | AI answers | 14% 96 | | Google-Extended https://www.crawlreadiness.com/crawlers/google-extended | Google AI | AI answers | 13.5% 92 | | cohere-ai https://www.crawlreadiness.com/crawlers/cohere-ai | Cohere | AI answers | 13.2% 90 | | Applebot-Extended https://www.crawlreadiness.com/crawlers/applebot-extended | Apple Intelligence | Training only | 12.9% 88 | | PetalBot https://www.crawlreadiness.com/crawlers/petalbot | Huawei | AI search index | 12.9% 88 | | Amazonbot https://www.crawlreadiness.com/crawlers/amazonbot | Amazon | AI answers | 12.3% 84 | | PerplexityBot https://www.crawlreadiness.com/crawlers/perplexitybot | Perplexity | AI search index | 12.3% 84 | Share of the 684 sites that answered whose robots.txt disallows / for the crawler, by name or through a group it falls under. Every crawler the check knows is in summary.json https://www.crawlreadiness.com/ai-access-index/2026-11/summary.json ; what each one is, in the crawler reference https://www.crawlreadiness.com/crawlers . The files and signals written for AI | Signal | What counts | Sites | |---|---|---| | llms.txt | a real one: a text file that starts with a Markdown heading | 15.8% 108 | | Content-Signal | a line in robots.txt saying what the content may be used for | 2.5% 17 | | TDM reservation | the EU text-and-data-mining opt-out, as a header or /.well-known/tdmrep.json | 0.9% 6 | | A markdown copy for agents | the homepage answers Accept: text/markdown with markdown | 3.7% 25 | | agents.json | at /.well-known/agents.json | 0.3% 2 | | A noai directive | in the homepage's meta robots or X-Robots-Tag | 0% 0 | Shares of the 684 sites that answered. Edition by edition | Edition | Answered | Open | AI search, no training | Selective | Closed | Refuse an allowed crawler | Serve a blocked one | |---|---|---|---|---|---|---|---| | November 2026 https://www.crawlreadiness.com/ai-access-index/2026-11 | 684 of 1000 | 79.7% | 8.8% | 3.8% | 7.7% | 11.3% | 63.7% | Method - The same sites every month. The top 1,000 hostnames of Tranco list 8PXYV https://tranco-list.eu/list/8PXYV , the research-standard ranking, downloaded on October 11, 2026 and frozen, so a change in the index is a change in the sites, not in the list. Some are infrastructure hostnames with no website behind them; they count as not answering. - The same day and the same rules. On the 1st of each month, from a GitHub Actions cloud runner this edition ran 13:27 to 13:37 UTC : each site once, six at a time, nothing retried. It is the check this site runs for anyone, scoring rules v3: robots.txt, llms.txt, the homepage and its markdown copy, agents.json and the TDM file as CrawlReadinessBot , then the homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, keeping only the status. A site's own report says what the index says about it. - The four policies. Read from robots.txt's verdicts for the 40 search and answer crawlers and the 9 training crawlers. Open to AI : at most a fifth of the search crawlers blocked and training left mostly open. AI search, no AI training : search open, training blocked or every training crawler the file names blocked . Closed to AI : every major AI company kept out. Selective : anything else, such as one company left open while the rest are blocked. - Refused and tested. A refusal is a 401, 403, 429 or 503. A crawler's refusal counts only when the homepage opened for the check's own request: when the server refused us too, it could be a wall against every bot. A 2xx to a crawler robots.txt blocks counts either way, so the share of sites that serve a blocked crawler includes some whose homepage did not open for us; the second figure on that tile, and the per-crawler table, leave them out. A site counts as tested when at least one crawler's answer can be read against its robots.txt. Each share names what it is a share of. - Limits. One pass, homepages only. The test sends a crawler's user-agent string, which is what firewall rules match; it does not come from the vendors' IP addresses, so a site that verifies crawlers by address could refuse us and admit the real crawler. Firewalls answer by network and by hour, and a 429 can be a rate limit, so the stricter figure leaves 429-only refusals out. The October 2026 study measured the same sites from a different network with a browser as its baseline, so its numbers are not part of this series. Download the data Quote the numbers anywhere, the press included, with a credit. The files are licensed CC BY-NC 4.0 https://creativecommons.org/licenses/by-nc/4.0/ : free to share and adapt for non-commercial use, with credit. Commercial use of the files, such as in a product or a dataset you sell, needs our permission: ask at support@crawlreadiness.com mailto:support@crawlreadiness.com . Cite as “Crawl Readiness, AI Access Index , November 2026, www.crawlreadiness.com/ai-access-index/2026-11.” - results.csv https://www.crawlreadiness.com/ai-access-index/2026-11/results.csv : one row per site in the sample, with its policy, the server's answer to each of the four crawlers and every signal. - changes.csv https://www.crawlreadiness.com/ai-access-index/2026-11/changes.csv : every change since the edition before, site by site. - summary.json https://www.crawlreadiness.com/ai-access-index/2026-11/summary.json : every number on this page, with what each is a share of. If you run a site in the sample and want its row out of the published files, write to support@crawlreadiness.com mailto:support@crawlreadiness.com from an address at that domain. Where does your site stand? The same check, free, with the fixes: run it on your site https://www.crawlreadiness.com/ . Sites people have checked with the free tool are listed, unranked, in Checked sites https://www.crawlreadiness.com/checked-sites .