A crawlability check often begins with a comforting result: request the homepage, receive HTTP 200, and declare the site open. That conclusion is too broad. The response only proves that one client, from one network, with one user agent and one request shape, reached one representation of the page at one moment.
AI crawlers do not necessarily share any of those conditions. A CDN may classify their user agents differently. A robots rule may apply to one bot but not another. A challenge page may return 200 while replacing the actual document. Even when the HTML is real, the canonical URL or robots metadata can make the page unusable for discovery.
The practical fix is to stop treating the status code as the verdict. Treat it as one item in an evidence chain.
Consider three requests to the same page. A normal browser user agent receives the product page. A generic script receives a managed challenge. A named crawler receives a 403 from a WAF rule. The URL has not changed, but the effective access policy has.
This is why a single curl command is a poor proxy for crawler access. It compresses transport, policy, identity, and content into one number. When the number is 200, it also hides soft failures: consent walls, bot challenges, login interstitials, and branded not-found pages can all be successful HTTP responses.
A useful probe records at least the requested user agent, status, final URL, content type, title, canonical URL, robots directives, and a small fingerprint of the returned body. The fingerprint does not need to retain the page. It only needs to show whether two clients received materially different documents.
async function probe(url, userAgent) {
const response = await fetch(url, {
redirect: "follow",
headers: { "user-agent": userAgent }
});
const html = await response.text();
const title = html.match(/<title[^>]*>([^<]*)<\/title>/i)?.[1] ?? null;
return {
status: response.status,
finalUrl: response.url,
contentType: response.headers.get("content-type"),
title,
bytes: Buffer.byteLength(html),
challenge: /just a moment|verify you are human/i.test(html)
};
}
The important property is not sophistication. It is comparability. Run the same probe with a baseline browser identity and each crawler identity, then explain the difference instead of guessing from one response.
Robots policy should be evaluated before interpreting a network response. Parse the groups for the exact bot token, apply the longest matching rule, and report which rule won. Do not assume that a wildcard group describes every named bot. Also keep robots decisions separate from firewall decisions: an allowed robots rule cannot override a CDN block, and a successful fetch does not erase a disallow rule.
Next, classify the transport outcome. DNS failure, timeout, TLS failure, 429, 403, and a Cloudflare challenge are different root causes. Lumping all of them into “blocked” makes the suggested repair unreliable. A timeout suggests availability or routing work. A 429 suggests pacing. A hard WAF block requires a rule review. A challenge means the probe did not obtain the target document, even if the status is 200.
Finally, verify page identity. Compare the final URL and canonical path with the requested page. Look for a plausible title and expected content markers. Detect obvious soft-404 titles. Read page-level robots metadata. Only after those checks should the system say that the crawler received the intended, indexable document.
I use this layered approach in a free AI crawler accessibility checker because a useful diagnostic should explain which gate failed, not merely display a red badge.
Some sites verify crawler IP ranges or signed requests in addition to the user agent. A public diagnostic cannot honestly impersonate those network identities. In that case, the correct result is not “allowed” or “blocked.” It is “the user-agent path passed, but origin verification was not tested.”
That distinction matters operationally. False confidence can leave important pages invisible, while false alarms encourage site owners to weaken WAF rules unnecessarily. Both are worse than a bounded answer.
The same principle applies to remediation. Recommend the smallest layer-specific change and preserve the evidence that justified it. If a robots group blocks a bot, show the matching lines. If a CDN returns a hard block, capture the classification and request identifier without storing cookies. If the page is a soft 404, show the title and canonical mismatch. A future recheck can then prove whether the specific failure changed.
A status code is still useful. It is simply not a conclusion. Crawlability is the conjunction of policy permission, network reachability, document identity, and page-level indexability. Measure those gates independently, preserve the first real cause, and leave uncertainty visible when the probe cannot test a provider-specific identity.