# AI crawler user agents are self-reported: 468 real fetches, 991 fake ones > Source: > Published: 2026-08-22 03:31:26+00:00 I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up. Then I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for `.env` and friends: 991. Of everything calling itself GPTBot, Cloudflare could verify 13% as OpenAI. Here is how to separate the impersonators on a Cloudflare free plan, and what the numbers looked like. Writing `GPTBot/1.2` into a header costs nothing, so a table grouped by UA is a list of claims. Behind Cloudflare there are two things to check those claims against: `verifiedBotCategory` What survives both filters is "verified bots fetching real pages", and that is the only number worth reporting. Add `userAgent` and `verifiedBotCategory` to the dimensions of `httpRequestsAdaptiveGroups` . `clientAsn` and `botClass` require a paid plan; `verifiedBotCategory` does not. ``` QUERY_BY_UA_VERIFIED = """ query ($zoneTag: String!, $since: Time!, $until: Time!) { viewer { zones(filter: { zoneTag: $zoneTag }) { httpRequestsAdaptiveGroups( limit: 5000 filter: { datetime_geq: $since, datetime_leq: $until } orderBy: [count_DESC] ) { count dimensions { userAgent verifiedBotCategory } } } } } """ ``` Two constraints to plan around: ``` python def fetch_rows(token, query, zone_tag, since, until): # split the range into one-day windows and concatenate the rows all_rows, cursor = [], since one_day = dt.timedelta(days=1) while cursor < until: win_end = min(cursor + one_day, until) variables = { "zoneTag": zone_tag, "since": cursor.isoformat() + "Z", "until": win_end.isoformat() + "Z", } try: all_rows.extend(rows_from(gql(token, query, variables, exit_on_error=False))) except RuntimeError as e: print(f" [skipped] {cursor.date()}-{win_end.date()}: {e}") # past retention cursor = win_end return all_rows ``` That single extra dimension is enough to split claim from reality. Eight days: | Claimed UA | Verified | Unverified | Verified share | |---|---|---|---| | ChatGPT-User | 211 (AI Assistant) | 336 | 39% | | Amazonbot | 135 (AI Crawler) | 489 | 22% | | ClaudeBot | 133 (AI Crawler) | 126 | 51% | | OAI-SearchBot | 61 (Search Engine Crawler) | 132 | 32% | | GPTBot | 19 (AI Crawler) | 123 | 13% | | meta-externalagent | 209 (AI Crawler) | 0 | 100% | | Applebot | 56 (AI Search) | 0 | 100% | | PerplexityBot | 0 | 144 | 0% | | Perplexity-User | 0 | 311 | 0% | Of 547 requests presenting as `ChatGPT-User` , 211 came from an address that traced back to OpenAI. Only two agents came through clean — `meta-externalagent` and `Applebot` , 100% verified with zero impersonation. Those are the only rows whose claimed totals are usable as-is. All 455 Perplexity-branded requests were unverified. 123 requests claimed `Google-Extended` . Verified share 0%, and 70 of them hit credential-scanning paths. No inference required. Google's [crawler documentation](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) states that `Google-Extended` has no separate HTTP request user agent string: crawling happens under the existing Google user agents, and the token exists purely to be addressed in robots.txt for AI-training control. So every request presenting that UA is, by definition, not Google. The same trick works for any operator that publishes IP ranges — OpenAI ships [gptbot.json](https://openai.com/gptbot.json). Verification alone isn't enough: a verified bot fetching robots.txt has read nothing. ``` SCAN_PATTERNS = ( "wp-", ".env", ".git", ".aws", ".svn", ".ssh", "secrets", "credentials", "config.json", "service_account", "actuator", "api/auth", "phpinfo", ".bak", ".yml", ".yaml", ".php", ".sql", "id_rsa", ".npmrc", ".htpasswd", ) OPS_PREFIXES = ("/robots.txt", "/sitemap", "/llms.txt", "/favicon", "/rss", "/feed", "/.well-known/") ASSET_PREFIXES = ("/_astro/", "/images/", "/assets/", "/fonts/", "/cdn-cgi/", "/_image") def classify_path(path, sitemap_paths): # content = a real page was consumed / ops = crawl bookkeeping # asset = static file / scan = credential probing / other = path does not exist if not path: return "other" low = path.lower() if any(k in low for k in SCAN_PATTERNS): return "scan" if low.startswith(OPS_PREFIXES): return "ops" if low.startswith(ASSET_PREFIXES): return "asset" if not sitemap_paths: return "unknown" # cannot assert existence, so cannot call it content return "content" if (path.rstrip("/") or "/") in sitemap_paths else "other" ``` Using the sitemap as the source of truth for existence is the part that holds up. Deciding from the response status looks easier, but redirects and paths that answer 200 without being real pages both leak in. The set of paths you declared public is a cleaner definition of "a page of mine". Keep the `unknown` branch too. Fold it into `content` and your numbers spike on any day the sitemap fetch fails, with nothing in the output to explain it. | Claimed UA | content | ops | scan | total | verified | |---|---|---|---|---|---| | ChatGPT-User | 216 | 0 | 181 | 547 | 39% | | meta-externalagent | 93 | 9 | 0 | 209 | 100% | | Amazonbot | 85 | 1 | 284 | 624 | 22% | | Applebot | 27 | 9 | 0 | 56 | 100% | | OAI-SearchBot | 17 | 43 | 74 | 193 | 32% | | PerplexityBot | 13 | 8 | 67 | 144 | 0% | | GPTBot | 7 | 9 | 72 | 142 | 13% | | ClaudeBot | 6 | 129 | 70 | 259 | 51% | | Google-Extended | 0 | 0 | 70 | 123 | 0% | | Perplexity-User | 0 | 0 | 173 | 311 | 0% | `content` totals 468, `scan` totals 991. By status code, 403s came to 957 against 705 served with 200. The ClaudeBot row is worth pulling apart: of 133 verified requests, 6 fetched articles and 129 fetched robots.txt and similar. "ClaudeBot sent 259 requests" and "six articles were read" are the same data. Keep the **verified share as a column** in whatever you output. When a series' share collapses, that is your signal to stop reading it as a metric for that snapshot. The week-over-week comparison (content 610 → 468 while scan went 357 → 991), why I cannot tell a genuine drop in interest apart from impersonation being reclassified, and the point where WAF blocks overtook served requests are all on Aulvem → [Aulvem | AI crawler user agents are self-reported: 468 real fetches, 991 fake ones](https://aulvem.com/blog/2026-08-22-ai-crawler-ua-verification/)