# robots.txt asks, the server answers. Method and data from the AI Access Index

> Source: <https://dev.to/grubbomatic/robotstxt-asks-the-server-answers-method-and-data-from-the-ai-access-index-98g>
> Published: 2026-11-18 22:58:06+00:00

**Key takeaway:** In a 1,000-site Tranco sample, 11.3% of sites allowing an AI crawler in robots.txt refused it with a 401/403/429/503, while 63.7% of sites blocking a crawler still returned a 2xx, so robots.txt alone does not control AI crawler access.

*Disclosure: we build Crawl Readiness, a website check that also sells paid audits and monitoring. We also maintain Machine Readiness, the standard Crawl Readiness measures against. No standards body or AI company has adopted it. Both are made by Tundrastone (Ontario, Canada).*

On 1 November 2026 we checked the top 1,000 hostnames of [Tranco list 8PXYV](https://tranco-list.eu/list/8PXYV) for edition 1 of the monthly [AI Access Index](https://www.crawlreadiness.com/ai-access-index/2026-11); 684 answered. Of the 547 tested sites that allow at least one of four AI crawlers in robots.txt (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot), 62 (11.3%) refused at least one crawler they allow, with a 401, 403, 429 or 503. Leaving out the 17 sites whose only refusals were 429s, it is 45 (8.2%). Of the 124 tested sites that block at least one of the four, 79 (63.7%) answered at least one crawler they block with a 2xx status.

Our test sends a user-agent string that names each crawler, from a cloud runner, not the vendors' IP addresses, so a site that verifies crawlers by address could refuse the test and admit the vendor's own crawler. The 11.3% counts sites that turned away a test request, so it is more exposed to that than the 63.7% is. A page served to a user-agent string does not show what the vendor's crawler would get.

robots.txt is a request, not a lock: it asks, and only the server, or a CDN or firewall in front of it, can refuse. A 2xx for a crawler that robots.txt asks to stay away breaks no rule, so we call it served, not a fault. In the standard we maintain, server enforcement is optional (A5, a MAY). Open or closed, a policy is a choice, and a block that holds is reported as working as intended.

Save [results.csv](https://www.crawlreadiness.com/ai-access-index/2026-11/results.csv) and this script as recount.mjs, then run `node recount.mjs results.csv`:

``` js
// split(",") is safe: only the last column holds commas.
import { readFileSync } from "node:fs";

const [head, ...lines] = readFileSync(process.argv[2], "utf8").trim().split("\n").map((l) => l.split(","));
const rows = lines.map((l) => Object.fromEntries(head.map((h, i) => [h, l[i]])));
const four = ["GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot"];
const has = (r, key, value) => four.some((c) => r[c + key] === value);

const answered = rows.filter((r) => r.status === "ok");
const tested = answered.filter((r) => r.integrity !== "unknown");
const allows = tested.filter((r) => has(r, "_robots", "allowed"));
const blocks = tested.filter((r) => has(r, "_robots", "blocked"));
const refused = allows.filter((r) => has(r, "_outcome", "refused-while-allowed"));
const refusedNot429 = refused.filter((r) => four.some((c) => r[c + "_outcome"] === "refused-while-allowed" && r[c + "_status"] !== "429"));
const served = blocks.filter((r) => has(r, "_outcome", "served-while-blocked"));
const opened = blocks.filter((r) => r.homepage_status.startsWith("2"));

console.log("answered", answered.length, "tested", tested.length);
console.log("allow base", allows.length, "refused", refused.length, "without 429-only", refusedNot429.length);
console.log("block base", blocks.length, "served", served.length);
console.log("own request opened", opened.length, "served", opened.filter((r) => served.includes(r)).length);
```

It prints 684 and 599, then 547, 62 and 45, then 124 and 79, then 91 and 74. In the `integrity` column, `both` (2 sites) counts in each, so 77 + 2 = 79 and 60 + 2 = 62. In summary.json the headline counts are under `server`. Each row also has a score and a grade. A site whose robots.txt blocks AI crawlers in a pattern the check reads as deliberate is graded Restricted by Design, a choice rather than a fault. Its blocked crawlers still cost points, so its score falls.

`rank` column is Tranco's; check Tranco's list page before reuse.
Crawl Readiness checks that a website is set up for AI the way its owner intends: readable by the AI crawlers it wants reading it, closed to the ones it wants closed, and with a server that keeps to what its robots.txt says. Readiness is not a promise of being mentioned. Corrections: comment here or write to [support@crawlreadiness.com](mailto:support@crawlreadiness.com).
