AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up.

Then I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for .env

and friends: 991. Of everything calling itself GPTBot, Cloudflare could verify 13% as OpenAI.

Here is how to separate the impersonators on a Cloudflare free plan, and what the numbers looked like.

Writing GPTBot/1.2

into a header costs nothing, so a table grouped by UA is a list of claims.

Behind Cloudflare there are two things to check those claims against:

verifiedBotCategory

What survives both filters is "verified bots fetching real pages", and that is the only number worth reporting.

Add userAgent

and verifiedBotCategory

to the dimensions of httpRequestsAdaptiveGroups

. clientAsn

and botClass

require a paid plan; verifiedBotCategory

does not.

QUERY_BY_UA_VERIFIED = """
query ($zoneTag: String!, $since: Time!, $until: Time!) {
  viewer {
    zones(filter: { zoneTag: $zoneTag }) {
      httpRequestsAdaptiveGroups(
        limit: 5000
        filter: { datetime_geq: $since, datetime_leq: $until }
        orderBy: [count_DESC]
      ) {
        count
        dimensions { userAgent verifiedBotCategory }
      }
    }
  }
}
"""

Two constraints to plan around:

def fetch_rows(token, query, zone_tag, since, until):
    all_rows, cursor = [], since
    one_day = dt.timedelta(days=1)
    while cursor < until:
        win_end = min(cursor + one_day, until)
        variables = {
            "zoneTag": zone_tag,
            "since": cursor.isoformat() + "Z",
            "until": win_end.isoformat() + "Z",
        }
        try:
            all_rows.extend(rows_from(gql(token, query, variables, exit_on_error=False)))
        except RuntimeError as e:
            print(f"    [skipped] {cursor.date()}-{win_end.date()}: {e}")   # past retention
        cursor = win_end
    return all_rows

That single extra dimension is enough to split claim from reality. Eight days:

Of 547 requests presenting as ChatGPT-User

, 211 came from an address that traced back to OpenAI.

Only two agents came through clean — meta-externalagent

and Applebot

, 100% verified with zero impersonation. Those are the only rows whose claimed totals are usable as-is. All 455 Perplexity-branded requests were unverified.

123 requests claimed Google-Extended

. Verified share 0%, and 70 of them hit credential-scanning paths.

No inference required. Google's crawler documentation states that Google-Extended

has no separate HTTP request user agent string: crawling happens under the existing Google user agents, and the token exists purely to be addressed in robots.txt for AI-training control.

So every request presenting that UA is, by definition, not Google. The same trick works for any operator that publishes IP ranges — OpenAI ships gptbot.json.

Verification alone isn't enough: a verified bot fetching robots.txt has read nothing.

SCAN_PATTERNS = (
    "wp-", ".env", ".git", ".aws", ".svn", ".ssh", "secrets", "credentials",
    "config.json", "service_account", "actuator", "api/auth", "phpinfo",
    ".bak", ".yml", ".yaml", ".php", ".sql", "id_rsa", ".npmrc", ".htpasswd",
)
OPS_PREFIXES = ("/robots.txt", "/sitemap", "/llms.txt", "/favicon", "/rss", "/feed", "/.well-known/")
ASSET_PREFIXES = ("/_astro/", "/images/", "/assets/", "/fonts/", "/cdn-cgi/", "/_image")

def classify_path(path, sitemap_paths):
    if not path:
        return "other"
    low = path.lower()
    if any(k in low for k in SCAN_PATTERNS):
        return "scan"
    if low.startswith(OPS_PREFIXES):
        return "ops"
    if low.startswith(ASSET_PREFIXES):
        return "asset"
    if not sitemap_paths:
        return "unknown"   # cannot assert existence, so cannot call it content
    return "content" if (path.rstrip("/") or "/") in sitemap_paths else "other"

Using the sitemap as the source of truth for existence is the part that holds up. Deciding from the response status looks easier, but redirects and paths that answer 200 without being real pages both leak in. The set of paths you declared public is a cleaner definition of "a page of mine".

Keep the unknown

branch too. Fold it into content

and your numbers spike on any day the sitemap fetch fails, with nothing in the output to explain it.

content

totals 468, scan

totals 991. By status code, 403s came to 957 against 705 served with 200.

The ClaudeBot row is worth pulling apart: of 133 verified requests, 6 fetched articles and 129 fetched robots.txt and similar. "ClaudeBot sent 259 requests" and "six articles were read" are the same data.

Keep the verified share as a column in whatever you output. When a series' share collapses, that is your signal to stop reading it as a metric for that snapshot.

The week-over-week comparison (content 610 → 468 while scan went 357 → 991), why I cannot tell a genuine drop in interest apart from impersonation being reclassified, and the point where WAF blocks overtook served requests are all on Aulvem → Aulvem | AI crawler user agents are self-reported: 468 real fetches, 991 fake ones