# How to Verify an AI Crawler Is Who It Says It Is

> Source: <https://dev.to/inxprncd/how-to-verify-an-ai-crawler-is-who-it-says-it-is-470e>
> Published: 2026-07-30 20:34:17+00:00

Your server logs show a request from `GPTBot`

. Should you trust it?

The honest answer is: not on the strength of that string alone. **A user-agent header is a claim the client makes about itself, and any client can claim anything.** `curl -A "GPTBot"`

takes about four seconds to type. If your robots.txt policy, your analytics, or your paywall logic branches on the user-agent string, it branches on unverified input.

The good news is that most major AI crawlers publish a way to check. The less good news is that a meaningful minority publish nothing at all, and one whole category is unverifiable by design. This post covers how each verification method works, which crawlers support which, and where the honest answer is still "you cannot tell."

Three things are worth separating:

robots.txt operates at layer three but has no access to layer two. It is an advisory protocol: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) standardises the syntax and the expectation of compliance, not a mechanism to enforce it. A crawler that ignores your `Disallow`

is not breaking a technical control; it is declining a request. And a crawler that *impersonates* a well-behaved one inherits whatever privileges you granted the original.

That matters more in 2026 than it did in 2020, because the privileges are now real. Sites allow AI crawlers to get represented in model answers, meter them under pay-per-crawl, or exempt them from bot challenges. Every one of those is a reason for someone to wear the costume.

The classic method, and still the most robust for operators who support it. You take the request's IP, resolve it to a hostname (`PTR`

record), check that the hostname ends in a domain the operator controls, then resolve that hostname *back* to an IP and confirm it matches the original.

The forward-confirm step is not optional. A `PTR`

record is set by whoever controls the IP block's reverse zone, so without resolving forward again you are trusting an attacker-controlled record.

Operators documenting a reverse-DNS suffix include Google (`.googlebot.com`

/ `.google.com`

), Microsoft (`.search.msn.com`

), Amazon (`.crawl.amazonbot.amazon`

), Apple (`.applebot.apple.com`

) and Common Crawl (`.crawl.commoncrawl.org`

).

The operator publishes a JSON file of the CIDR blocks its crawler uses; you check membership. OpenAI does this at `openai.com/gptbot.json`

, and it is the most common method in practice — **20 of the 41 crawlers in our registry document IP-range verification, making it the single most widely supported check.**

Two operational caveats. The list changes, so fetch and cache it on a schedule rather than hardcoding; and ranges are only as trustworthy as the operator's own infrastructure hygiene — a shared cloud range proves less than a dedicated one.

The direction of travel, and the only method that proves identity cryptographically rather than by network topology. The bot signs its HTTP requests with a private key using HTTP Message Signatures, publishes the public key in a discoverable JWKS directory, and your server verifies the signature. It is progressing through the IETF, and Cloudflare, Akamai and AWS have shipped support on the verifier side.

Here is the honest status from the operator side, which is the part most write-ups skip: **checking the primary documentation of all 41 crawlers in our registry, we could not confirm a single one that publishes a signature-agent domain and JWKS URL you can verify against today.** The infrastructure exists; the operator-side rollout is not yet documented in a way a site owner can act on. Treat Web Bot Auth as the thing to build toward, not the thing to gate on this quarter.

Counting verification methods across the registry (a crawler can support more than one):

| Method | Crawlers documenting it |
|---|---|
| Published IP range | 20 |
| Reverse DNS | 11 |
| IP signature | 4 |
No network-level method at all |
16 |

A crawler can document more than one method, so the first three rows overlap: **25 of 41 support at least one network-level check, and the remaining 16 support none.** For those 16 the user-agent string is the only identifier the operator publishes, and there is nothing to check it against. You can allow them, you can block them, but you cannot verify them. (One of the 16, `Google-Extended`

, has not even a user-agent string — see below.)

By purpose, the same 41 break down as: 13 user-triggered fetchers (an agent fetching a page because a human just asked), 10 search-index crawlers, 9 training crawlers, 4 agentic browsers, 3 data providers, and 2 pure opt-out tokens — `Google-Extended`

and `Applebot-Extended`

are not crawlers at all but robots.txt tokens that control how an already-fetched page may be used.

That distinction is worth internalising before you write a single `Disallow`

. Blocking a training crawler costs you nothing in referral traffic. Blocking a user-triggered fetcher means a human asked their assistant about your page and the assistant could not read it.

Agentic browsers are a genuinely different problem. They are real browser engines driven by an AI agent on a user's behalf. They carry no stable bot token, because from the network's point of view they *are* a browser: real Chrome user-agent, real TLS fingerprint, real JavaScript execution.

The registry tracks four, and the churn in that list is itself informative: ChatGPT Atlas (OpenAI) and Comet (Perplexity) are current, while OpenAI's Operator and Google's Project Mariner are already recorded as deprecated — superseded inside eighteen months. Whatever you build here, build it to be replaced.

There is no user-agent string to match, no published IP range, no reverse DNS suffix. Behavioural signals are all that remain, and behavioural signals on a headless-but-real browser are weak.

This is the honest edge of the current toolset, and it is why Web Bot Auth matters: it is the only proposal that would let an agentic browser *volunteer* a verifiable identity rather than requiring you to infer one.

Reverse DNS with forward confirmation, in Node with no dependencies:

``` js
import { promises as dns } from "node:dns";

// Suffixes an operator has published for its crawler. Never infer these —
// use the domain the operator documents, or you will validate an impostor.
const SUFFIXES = {
  googlebot: [".googlebot.com", ".google.com"],
  bingbot:   [".search.msn.com"],
  applebot:  [".applebot.apple.com"],
  amazonbot: [".crawl.amazonbot.amazon"],
  ccbot:     [".crawl.commoncrawl.org"],
};

export async function verifyByReverseDns(ip, operator) {
  const suffixes = SUFFIXES[operator];
  if (!suffixes) return { verified: false, reason: "no published suffix" };

  let hostnames;
  try {
    hostnames = await dns.reverse(ip);            // PTR lookup
  } catch {
    return { verified: false, reason: "no PTR record" };
  }

  const host = hostnames.find((h) => suffixes.some((s) => h.endsWith(s)));
  if (!host) return { verified: false, reason: "PTR outside published domain" };

  // Forward-confirm: the PTR record alone is controlled by whoever owns the
  // reverse zone, so resolve the hostname back and require the IP to match.
  const forward = await dns.resolve(host).catch(() => []);
  return forward.includes(ip)
    ? { verified: true, host }
    : { verified: false, reason: "forward lookup did not match" };
}
```

Two things to get right in production. Do this **out of band** — a DNS round trip inside the request path is a latency and denial-of-service liability, so verify asynchronously and cache the verdict per IP. And **fail closed on your policy, not on the request**: an unverified request is not necessarily hostile, it is merely unproven, so downgrade its privileges rather than returning a 403 to what might be a real user.

Most published AI-crawler lists give every bot a clean yes/no for robots.txt compliance. Compiling this data, that cleanliness turned out to be the least defensible part.

For a good number of crawlers, the operator has simply never published an explicit statement about the agent under that exact name. The available options are then to infer from a sibling bot, to copy what another list asserts, or to record "unclear" and cite what the operator actually says. Only the third survives contact with someone who checks.

So the registry behind this post carries a tri-state — yes, no, unclear — with a primary source URL attached to each individual claim and a `last_verified`

date, plus explicit nulls where a field could not be sourced rather than a plausible-looking guess. It is less tidy. It is the only version that stays true when an operator quietly changes their docs.

If you want the per-crawler data — user-agent tokens, robots.txt tokens, verification methods, published IP-range URLs, opt-out mechanisms and the source URL behind every field — it is at [agentswelcome.dev/crawlers](https://agentswelcome.dev/crawlers), and there is a JSON endpoint at `/api/crawlers`

if you would rather diff it than read it.

**Does blocking AI crawlers in robots.txt actually stop them?**

Not technically. robots.txt is advisory: RFC 9309 defines the syntax and the expectation, not an enforcement mechanism. OpenAI, Anthropic and Perplexity each publish documentation stating their crawlers respect it, and the evidence is that the major operators do. Enforcement, if you need it, is a WAF rule or a verified-bot policy at your edge — not a text file.

**Is Google-Extended a crawler I should block?**

`Google-Extended`

is not a crawler and has no user-agent string. It is a robots.txt token controlling whether content Google already fetched may be used for Gemini training. Disallowing it does not reduce crawl traffic and does not affect Search indexing. `Applebot-Extended`

works the same way for Apple.**Should I block training crawlers but allow search crawlers?**

That is the common policy, and the reasoning holds: a training crawler consumes your content with no referral path back, while a search-index or user-triggered fetcher can surface you in an answer a human is reading right now. Decide per bot type rather than per company — several operators run both kinds under different tokens.

**How often does this data change?**

Often enough to matter. Operators add tokens, rename agents and update IP ranges without announcement. Any crawler list without a per-claim source URL and a verification date is a snapshot of someone's afternoon, including this one — which is why the numbers above carry the date they were checked: 2026-07-30.

*Data from the AGENTS WELCOME crawler registry — 41 crawlers, each field carrying its own primary source and verification date. Checked 2026-07-30.*