{"slug": "robots-txt-asks-the-server-answers-method-and-data-from-the-ai-access-index", "title": "robots.txt asks, the server answers. Method and data from the AI Access Index", "summary": "Tundrastone, the Ontario company behind the Crawl Readiness website check, published edition 1 of its AI Access Index on 1 November 2026, testing the top 1,000 Tranco hostnames (684 responded) against four AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Of 547 tested sites that allow at least one crawler in robots.txt, 62 (11.3%) refused at least one allowed crawler with a 401, 403, 429 or 503, dropping to 45 (8.2%) when 429-only refusals are excluded; of 124 sites blocking at least one crawler, 79 (63.7%) still served a blocked crawler with a 2xx status. The company notes the test sends user-agent strings from a cloud runner rather than vendor IP addresses, and that robots.txt is a request that only the server or an intermediary can enforce.", "body_md": "*Disclosure: we build Crawl Readiness, a website check that also sells paid audits and monitoring. We also maintain Machine Readiness, the standard Crawl Readiness measures against. No standards body or AI company has adopted it. Both are made by Tundrastone (Ontario, Canada).*\n\nOn 1 November 2026 we checked the top 1,000 hostnames of [Tranco list 8PXYV](https://tranco-list.eu/list/8PXYV) for edition 1 of the monthly [AI Access Index](https://www.crawlreadiness.com/ai-access-index/2026-11); 684 answered. Of the 547 tested sites that allow at least one of four AI crawlers in robots.txt (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot), 62 (11.3%) refused at least one crawler they allow, with a 401, 403, 429 or 503. Leaving out the 17 sites whose only refusals were 429s, it is 45 (8.2%). Of the 124 tested sites that block at least one of the four, 79 (63.7%) answered at least one crawler they block with a 2xx status.\n\nOur test sends a user-agent string that names each crawler, from a cloud runner, not the vendors' IP addresses, so a site that verifies crawlers by address could refuse the test and admit the vendor's own crawler. The 11.3% counts sites that turned away a test request, so it is more exposed to that than the 63.7% is. A page served to a user-agent string does not show what the vendor's crawler would get.\n\nrobots.txt is a request, not a lock: it asks, and only the server, or a CDN or firewall in front of it, can refuse. A 2xx for a crawler that robots.txt asks to stay away breaks no rule, so we call it served, not a fault. In the standard we maintain, server enforcement is optional (A5, a MAY). Open or closed, a policy is a choice, and a block that holds is reported as working as intended.\n\nSave [results.csv](https://www.crawlreadiness.com/ai-access-index/2026-11/results.csv) and this script as recount.mjs, then run `node recount.mjs results.csv`:\n\n``` js\n// split(\",\") is safe: only the last column holds commas.\nimport { readFileSync } from \"node:fs\";\n\nconst [head, ...lines] = readFileSync(process.argv[2], \"utf8\").trim().split(\"\\n\").map((l) => l.split(\",\"));\nconst rows = lines.map((l) => Object.fromEntries(head.map((h, i) => [h, l[i]])));\nconst four = [\"GPTBot\", \"OAI-SearchBot\", \"ClaudeBot\", \"PerplexityBot\"];\nconst has = (r, key, value) => four.some((c) => r[c + key] === value);\n\nconst answered = rows.filter((r) => r.status === \"ok\");\nconst tested = answered.filter((r) => r.integrity !== \"unknown\");\nconst allows = tested.filter((r) => has(r, \"_robots\", \"allowed\"));\nconst blocks = tested.filter((r) => has(r, \"_robots\", \"blocked\"));\nconst refused = allows.filter((r) => has(r, \"_outcome\", \"refused-while-allowed\"));\nconst refusedNot429 = refused.filter((r) => four.some((c) => r[c + \"_outcome\"] === \"refused-while-allowed\" && r[c + \"_status\"] !== \"429\"));\nconst served = blocks.filter((r) => has(r, \"_outcome\", \"served-while-blocked\"));\nconst opened = blocks.filter((r) => r.homepage_status.startsWith(\"2\"));\n\nconsole.log(\"answered\", answered.length, \"tested\", tested.length);\nconsole.log(\"allow base\", allows.length, \"refused\", refused.length, \"without 429-only\", refusedNot429.length);\nconsole.log(\"block base\", blocks.length, \"served\", served.length);\nconsole.log(\"own request opened\", opened.length, \"served\", opened.filter((r) => served.includes(r)).length);\n```\n\nIt prints 684 and 599, then 547, 62 and 45, then 124 and 79, then 91 and 74. In the `integrity` column, `both` (2 sites) counts in each, so 77 + 2 = 79 and 60 + 2 = 62. In summary.json the headline counts are under `server`. Each row also has a score and a grade. A site whose robots.txt blocks AI crawlers in a pattern the check reads as deliberate is graded Restricted by Design, a choice rather than a fault. Its blocked crawlers still cost points, so its score falls.\n\n`rank` column is Tranco's; check Tranco's list page before reuse.\nCrawl Readiness checks that a website is set up for AI the way its owner intends: readable by the AI crawlers it wants reading it, closed to the ones it wants closed, and with a server that keeps to what its robots.txt says. Readiness is not a promise of being mentioned. Corrections: comment here or write to [support@crawlreadiness.com](mailto:support@crawlreadiness.com).", "url": "https://wpnews.pro/news/robots-txt-asks-the-server-answers-method-and-data-from-the-ai-access-index", "canonical_source": "https://dev.to/grubbomatic/robotstxt-asks-the-server-answers-method-and-data-from-the-ai-access-index-98g", "published_at": "2026-11-18 22:58:06+00:00", "updated_at": "2026-11-18 23:11:52.435099+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search"], "entities": ["Tundrastone", "Crawl Readiness", "AI Access Index", "Tranco", "GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot"], "also_reported_by": [], "takeaway": "In a 1,000-site Tranco sample, 11.3% of sites allowing an AI crawler in robots.txt refused it with a 401/403/429/503, while 63.7% of sites blocking a crawler still returned a 2xx, so robots.txt alone does not control AI crawler access.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/robots-txt-asks-the-server-answers-method-and-data-from-the-ai-access-index", "markdown": "https://wpnews.pro/news/robots-txt-asks-the-server-answers-method-and-data-from-the-ai-access-index.md", "text": "https://wpnews.pro/news/robots-txt-asks-the-server-answers-method-and-data-from-the-ai-access-index.txt", "jsonld": "https://wpnews.pro/news/robots-txt-asks-the-server-answers-method-and-data-from-the-ai-access-index.jsonld"}}