robots.txt asks, the server answers. Method and data from the AI Access Index Tundrastone, the Ontario company behind the Crawl Readiness website check, published edition 1 of its AI Access Index on 1 November 2026, testing the top 1,000 Tranco hostnames (684 responded) against four AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Of 547 tested sites that allow at least one crawler in robots.txt, 62 (11.3%) refused at least one allowed crawler with a 401, 403, 429 or 503, dropping to 45 (8.2%) when 429-only refusals are excluded; of 124 sites blocking at least one crawler, 79 (63.7%) still served a blocked crawler with a 2xx status. The company notes the test sends user-agent strings from a cloud runner rather than vendor IP addresses, and that robots.txt is a request that only the server or an intermediary can enforce. Disclosure: we build Crawl Readiness, a website check that also sells paid audits and monitoring. We also maintain Machine Readiness, the standard Crawl Readiness measures against. No standards body or AI company has adopted it. Both are made by Tundrastone Ontario, Canada . On 1 November 2026 we checked the top 1,000 hostnames of Tranco list 8PXYV https://tranco-list.eu/list/8PXYV for edition 1 of the monthly AI Access Index https://www.crawlreadiness.com/ai-access-index/2026-11 ; 684 answered. Of the 547 tested sites that allow at least one of four AI crawlers in robots.txt GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot , 62 11.3% refused at least one crawler they allow, with a 401, 403, 429 or 503. Leaving out the 17 sites whose only refusals were 429s, it is 45 8.2% . Of the 124 tested sites that block at least one of the four, 79 63.7% answered at least one crawler they block with a 2xx status. Our test sends a user-agent string that names each crawler, from a cloud runner, not the vendors' IP addresses, so a site that verifies crawlers by address could refuse the test and admit the vendor's own crawler. The 11.3% counts sites that turned away a test request, so it is more exposed to that than the 63.7% is. A page served to a user-agent string does not show what the vendor's crawler would get. robots.txt is a request, not a lock: it asks, and only the server, or a CDN or firewall in front of it, can refuse. A 2xx for a crawler that robots.txt asks to stay away breaks no rule, so we call it served, not a fault. In the standard we maintain, server enforcement is optional A5, a MAY . Open or closed, a policy is a choice, and a block that holds is reported as working as intended. Save results.csv https://www.crawlreadiness.com/ai-access-index/2026-11/results.csv and this script as recount.mjs, then run node recount.mjs results.csv : js // split "," is safe: only the last column holds commas. import { readFileSync } from "node:fs"; const head, ...lines = readFileSync process.argv 2 , "utf8" .trim .split "\n" .map l = l.split "," ; const rows = lines.map l = Object.fromEntries head.map h, i = h, l i ; const four = "GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot" ; const has = r, key, value = four.some c = r c + key === value ; const answered = rows.filter r = r.status === "ok" ; const tested = answered.filter r = r.integrity == "unknown" ; const allows = tested.filter r = has r, " robots", "allowed" ; const blocks = tested.filter r = has r, " robots", "blocked" ; const refused = allows.filter r = has r, " outcome", "refused-while-allowed" ; const refusedNot429 = refused.filter r = four.some c = r c + " outcome" === "refused-while-allowed" && r c + " status" == "429" ; const served = blocks.filter r = has r, " outcome", "served-while-blocked" ; const opened = blocks.filter r = r.homepage status.startsWith "2" ; console.log "answered", answered.length, "tested", tested.length ; console.log "allow base", allows.length, "refused", refused.length, "without 429-only", refusedNot429.length ; console.log "block base", blocks.length, "served", served.length ; console.log "own request opened", opened.length, "served", opened.filter r = served.includes r .length ; It prints 684 and 599, then 547, 62 and 45, then 124 and 79, then 91 and 74. In the integrity column, both 2 sites counts in each, so 77 + 2 = 79 and 60 + 2 = 62. In summary.json the headline counts are under server . Each row also has a score and a grade. A site whose robots.txt blocks AI crawlers in a pattern the check reads as deliberate is graded Restricted by Design, a choice rather than a fault. Its blocked crawlers still cost points, so its score falls. rank column is Tranco's; check Tranco's list page before reuse. Crawl Readiness checks that a website is set up for AI the way its owner intends: readable by the AI crawlers it wants reading it, closed to the ones it wants closed, and with a server that keeps to what its robots.txt says. Readiness is not a promise of being mentioned. Corrections: comment here or write to support@crawlreadiness.com mailto:support@crawlreadiness.com .