cd /news/ai-crawlers/ai-access-index-nov-2026-how-684-of-… · home › topics › ai-crawlers › article
[ARTICLE · art-172174] src=crawlreadiness.com ↗ pub= topic=ai-crawlers verified=true sentiment=· neutral

AI Access Index, Nov 2026: how 684 of the topk hostnames treat AI crawlers

The first AI Access Index, published November 1, 2026, found that 79.7% (545) of the 684 top-1,000 hostnames that answered are open to AI crawlers for both search and training, while 7.7% (53) block AI entirely and 8.8% (60) allow AI search but not training. The index also measured server behavior against robots.txt, finding that 75% of the 80 sites blocking GPTBot in robots.txt still served its requests, and Bytespider (Bytedance) was the most-blocked crawler at 17.5% (120 sites), followed by CCBot at 17.3% (118) and GPTBot at 15.6% (107).

key takeaway

Across 684 top hostnames, robots.txt blocking of GPTBot (15.6%) exceeded server-level refusal (8.8%), and 75% of the 80 sites that block GPTBot in robots.txt still served its requests.

read7 min views7 publishedNov 18, 2026
AI Access Index, Nov 2026: how 684 of the topk hostnames treat AI crawlers
Image: source

On the 1st of every month, the same 1,000 sites, checked the same way: what each site's robots.txt tells AI crawlers, and what its server actually does when they arrive. It is not a ranking. Open or closed, a policy is a choice; the index measures which choices sites make, and whether their servers keep to them.

What robots.txt tells AI crawlers #

Each site's policy is read the way its own report reads it: from what its robots.txt says to the 40 crawlers that fetch pages for AI search and answers, and the 9 that gather training data.

- Open to AI79.7% (545)AI search: yes · AI training: yes
- AI search, no AI training8.8% (60)AI search: yes · AI training: no
- Selective3.8% (26)Some AI companies in, others kept out
- Closed to AI7.7% (53)AI search: no · AI training: no

Shares of the 684 sites that answered. Open includes 119 sites with no robots.txt at all, which allows everything. Closed includes 13 hosts that close themselves to every crawler, search engines included: mostly API and CDN hostnames rather than a decision about AI.

Does the server do what robots.txt says? #

robots.txt is a request, and the server answers first. After the check's own visit, each homepage was asked for again four times, each time with one AI crawler's user-agent string, and the answer was set beside what the site's robots.txt says about that crawler.

Crawler Blocked in robots.txt Refused by the server Refused while robots.txt allows it Served while robots.txt blocks it
GPTBot OpenAI training 15.6% 8.8% 5.9% 75% of 80
OAI-SearchBot ChatGPT search 8.2% 3.8% 2.3% 76.1% of 46
ClaudeBot Anthropic training 15.5% 8.7% 5.5% 74.7% of 75
PerplexityBot Perplexity search 12.3% 8.5% 6.3% 71.7% of 60

Blocked in robots.txt: share of the 684 sites that answered. The server columns: share of the sites whose homepage opened for the check's own request (about 525). Served while blocked: share of those that block that crawler. A refusal is a 401, 403, 429 or 503.

What changed since last month #

This is the first edition. From the December 2026 edition on, this section names every site that changed its AI policy, started or stopped blocking one of the four crawlers, or whose server started or stopped refusing a crawler its robots.txt allows.

Look up a site #

Every site in the index, as it answered on November 1, 2026. A site that is not in the sample can be checked the same way, free.

The most-blocked AI crawlers #

| Crawler | Company | What it does | Blocked by |

|---|---|---|---|
| [`Bytespider`](https://www.crawlreadiness.com/crawlers/bytespider) | Bytedance | Training only | 17.5% (120) | 
| [`CCBot`](https://www.crawlreadiness.com/crawlers/ccbot) | Common Crawl | Training only | 17.3% (118) | 
| [`GPTBot`](https://www.crawlreadiness.com/crawlers/gptbot) | OpenAI | Training only | 15.6% (107) | 
| [`ClaudeBot`](https://www.crawlreadiness.com/crawlers/claudebot) | Anthropic | Training only | 15.5% (106) | 
| [`Diffbot`](https://www.crawlreadiness.com/crawlers/diffbot) | Diffbot | AI answers | 14% (96) | 
| [`meta-externalagent`](https://www.crawlreadiness.com/crawlers/meta-externalagent) | Meta AI | AI answers | 14% (96) | 
| [`Google-Extended`](https://www.crawlreadiness.com/crawlers/google-extended) | Google AI | AI answers | 13.5% (92) | 
| [`cohere-ai`](https://www.crawlreadiness.com/crawlers/cohere-ai) | Cohere | AI answers | 13.2% (90) | 
| [`Applebot-Extended`](https://www.crawlreadiness.com/crawlers/applebot-extended) | Apple Intelligence | Training only | 12.9% (88) | 
| [`PetalBot`](https://www.crawlreadiness.com/crawlers/petalbot) | Huawei | AI search index | 12.9% (88) | 
| [`Amazonbot`](https://www.crawlreadiness.com/crawlers/amazonbot) | Amazon | AI answers | 12.3% (84) | 
| [`PerplexityBot`](https://www.crawlreadiness.com/crawlers/perplexitybot) | Perplexity | AI search index | 12.3% (84) | 

Share of the 684 sites that answered whose robots.txt disallows / for the crawler, by name or through a group it falls under. Every crawler the check knows is in summary.json; what each one is, in the crawler reference.

The files and signals written for AI #

Signal What counts Sites
llms.txt a real one: a text file that starts with a Markdown heading 15.8% (108)
Content-Signal a line in robots.txt saying what the content may be used for 2.5% (17)
TDM reservation the EU text-and-data-mining opt-out, as a header or /.well-known/tdmrep.json 0.9% (6)
A markdown copy for agents the homepage answers Accept: text/markdown with markdown 3.7% (25)
agents.json at /.well-known/agents.json 0.3% (2)
A noai directive in the homepage's meta robots or X-Robots-Tag 0% (0)

Shares of the 684 sites that answered.

Edition by edition #

| Edition | Answered | Open | AI search, no training | Selective | Closed | Refuse an allowed crawler | Serve a blocked one |

|---|---|---|---|---|---|---|---|
| [November 2026](https://www.crawlreadiness.com/ai-access-index/2026-11) | 684 of 1000 | 79.7% | 8.8% | 3.8% | 7.7% | 11.3% | 63.7% | 

Method #

  • The same sites every month. The top 1,000 hostnames ofTranco list 8PXYV , the research-standard ranking, downloaded on October 11, 2026 and frozen, so a change in the index is a change in the sites, not in the list. Some are infrastructure hostnames with no website behind them; they count as not answering.
  • The same day and the same rules. On the 1st of each month, from a GitHub Actions cloud runner (this edition ran 13:27 to 13:37 UTC): each site once, six at a time, nothing retried. It is the check this site runs for anyone, scoring rules v3: robots.txt, llms.txt, the homepage and its markdown copy, agents.json and the TDM file asCrawlReadinessBot , then the homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, keeping only the status. A site's own report says what the index says about it.
  • The four policies. Read from robots.txt's verdicts for the 40 search and answer crawlers and the 9 training crawlers.Open to AI : at most a fifth of the search crawlers blocked and training left mostly open.AI search, no AI training : search open, training blocked (or every training crawler the file names blocked).Closed to AI : every major AI company kept out.Selective : anything else, such as one company left open while the rest are blocked.
  • Refused and tested. A refusal is a 401, 403, 429 or 503. A crawler's refusal counts only when the homepage opened for the check's own request: when the server refused us too, it could be a wall against every bot. A 2xx to a crawler robots.txt blocks counts either way, so the share of sites that serve a blocked crawler includes some whose homepage did not open for us; the second figure on that tile, and the per-crawler table, leave them out. A site counts as tested when at least one crawler's answer can be read against its robots.txt. Each share names what it is a share of.
  • Limits. One pass, homepages only. The test sends a crawler's user-agent string, which is what firewall rules match; it does not come from the vendors' IP addresses, so a site that verifies crawlers by address could refuse us and admit the real crawler. Firewalls answer by network and by hour, and a 429 can be a rate limit, so the stricter figure leaves 429-only refusals out. The October 2026 study measured the same sites from a different network with a browser as its baseline, so its numbers are not part of this series.

Download the data #

Quote the numbers anywhere, the press included, with a credit. The files are licensed CC BY-NC 4.0: free to share and adapt for non-commercial use, with credit. Commercial use of the files, such as in a product or a dataset you sell, needs our permission: ask at support@crawlreadiness.com. Cite as “Crawl Readiness, AI Access Index, November 2026, www.crawlreadiness.com/ai-access-index/2026-11.”

  • results.csv : one row per site in the sample, with its policy, the server's answer to each of the four crawlers and every signal.
- [changes.csv](https://www.crawlreadiness.com/ai-access-index/2026-11/changes.csv) : every change since the edition before, site by site.
- [summary.json](https://www.crawlreadiness.com/ai-access-index/2026-11/summary.json) : every number on this page, with what each is a share of.

If you run a site in the sample and want its row out of the published files, write to [support@crawlreadiness.com](mailto:support@crawlreadiness.com) from an address at that domain.

Where does your site stand? #

The same check, free, with the fixes: run it on your site. Sites people have checked with the free tool are listed, unranked, in Checked sites.

── more in #ai-crawlers 4 stories · sorted by recency
── more on @ai access index 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/ai-access-index-nov-…] indexed:0 read:7min 2026-11-18 · —