AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

A developer found that most AI crawler traffic on their site is fake, with only 468 of 1,459 requests actually fetching articles. Using Cloudflare's verifiedBotCategory, they showed that only 13% of GPTBot claims were verified, while meta-externalagent and Applebot were 100% verified. The developer provides a method to distinguish real AI crawlers from impersonators using free Cloudflare tools.

I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up. Then I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for .env and friends: 991. Of everything calling itself GPTBot, Cloudflare could verify 13% as OpenAI. Here is how to separate the impersonators on a Cloudflare free plan, and what the numbers looked like. Writing GPTBot/1.2 into a header costs nothing, so a table grouped by UA is a list of claims. Behind Cloudflare there are two things to check those claims against: verifiedBotCategory What survives both filters is "verified bots fetching real pages", and that is the only number worth reporting. Add userAgent and verifiedBotCategory to the dimensions of httpRequestsAdaptiveGroups . clientAsn and botClass require a paid plan; verifiedBotCategory does not. QUERY BY UA VERIFIED = """ query $zoneTag: String , $since: Time , $until: Time { viewer { zones filter: { zoneTag: $zoneTag } { httpRequestsAdaptiveGroups limit: 5000 filter: { datetime geq: $since, datetime leq: $until } orderBy: count DESC { count dimensions { userAgent verifiedBotCategory } } } } } """ Two constraints to plan around: python def fetch rows token, query, zone tag, since, until : split the range into one-day windows and concatenate the rows all rows, cursor = , since one day = dt.timedelta days=1 while cursor < until: win end = min cursor + one day, until variables = { "zoneTag": zone tag, "since": cursor.isoformat + "Z", "until": win end.isoformat + "Z", } try: all rows.extend rows from gql token, query, variables, exit on error=False except RuntimeError as e: print f" skipped {cursor.date }-{win end.date }: {e}" past retention cursor = win end return all rows That single extra dimension is enough to split claim from reality. Eight days: | Claimed UA | Verified | Unverified | Verified share | |---|---|---|---| | ChatGPT-User | 211 AI Assistant | 336 | 39% | | Amazonbot | 135 AI Crawler | 489 | 22% | | ClaudeBot | 133 AI Crawler | 126 | 51% | | OAI-SearchBot | 61 Search Engine Crawler | 132 | 32% | | GPTBot | 19 AI Crawler | 123 | 13% | | meta-externalagent | 209 AI Crawler | 0 | 100% | | Applebot | 56 AI Search | 0 | 100% | | PerplexityBot | 0 | 144 | 0% | | Perplexity-User | 0 | 311 | 0% | Of 547 requests presenting as ChatGPT-User , 211 came from an address that traced back to OpenAI. Only two agents came through clean — meta-externalagent and Applebot , 100% verified with zero impersonation. Those are the only rows whose claimed totals are usable as-is. All 455 Perplexity-branded requests were unverified. 123 requests claimed Google-Extended . Verified share 0%, and 70 of them hit credential-scanning paths. No inference required. Google's crawler documentation https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers states that Google-Extended has no separate HTTP request user agent string: crawling happens under the existing Google user agents, and the token exists purely to be addressed in robots.txt for AI-training control. So every request presenting that UA is, by definition, not Google. The same trick works for any operator that publishes IP ranges — OpenAI ships gptbot.json https://openai.com/gptbot.json . Verification alone isn't enough: a verified bot fetching robots.txt has read nothing. SCAN PATTERNS = "wp-", ".env", ".git", ".aws", ".svn", ".ssh", "secrets", "credentials", "config.json", "service account", "actuator", "api/auth", "phpinfo", ".bak", ".yml", ".yaml", ".php", ".sql", "id rsa", ".npmrc", ".htpasswd", OPS PREFIXES = "/robots.txt", "/sitemap", "/llms.txt", "/favicon", "/rss", "/feed", "/.well-known/" ASSET PREFIXES = "/ astro/", "/images/", "/assets/", "/fonts/", "/cdn-cgi/", "/ image" def classify path path, sitemap paths : content = a real page was consumed / ops = crawl bookkeeping asset = static file / scan = credential probing / other = path does not exist if not path: return "other" low = path.lower if any k in low for k in SCAN PATTERNS : return "scan" if low.startswith OPS PREFIXES : return "ops" if low.startswith ASSET PREFIXES : return "asset" if not sitemap paths: return "unknown" cannot assert existence, so cannot call it content return "content" if path.rstrip "/" or "/" in sitemap paths else "other" Using the sitemap as the source of truth for existence is the part that holds up. Deciding from the response status looks easier, but redirects and paths that answer 200 without being real pages both leak in. The set of paths you declared public is a cleaner definition of "a page of mine". Keep the unknown branch too. Fold it into content and your numbers spike on any day the sitemap fetch fails, with nothing in the output to explain it. | Claimed UA | content | ops | scan | total | verified | |---|---|---|---|---|---| | ChatGPT-User | 216 | 0 | 181 | 547 | 39% | | meta-externalagent | 93 | 9 | 0 | 209 | 100% | | Amazonbot | 85 | 1 | 284 | 624 | 22% | | Applebot | 27 | 9 | 0 | 56 | 100% | | OAI-SearchBot | 17 | 43 | 74 | 193 | 32% | | PerplexityBot | 13 | 8 | 67 | 144 | 0% | | GPTBot | 7 | 9 | 72 | 142 | 13% | | ClaudeBot | 6 | 129 | 70 | 259 | 51% | | Google-Extended | 0 | 0 | 70 | 123 | 0% | | Perplexity-User | 0 | 0 | 173 | 311 | 0% | content totals 468, scan totals 991. By status code, 403s came to 957 against 705 served with 200. The ClaudeBot row is worth pulling apart: of 133 verified requests, 6 fetched articles and 129 fetched robots.txt and similar. "ClaudeBot sent 259 requests" and "six articles were read" are the same data. Keep the verified share as a column in whatever you output. When a series' share collapses, that is your signal to stop reading it as a metric for that snapshot. The week-over-week comparison content 610 → 468 while scan went 357 → 991 , why I cannot tell a genuine drop in interest apart from impersonation being reclassified, and the point where WAF blocks overtook served requests are all on Aulvem → Aulvem | AI crawler user agents are self-reported: 468 real fetches, 991 fake ones https://aulvem.com/blog/2026-08-22-ai-crawler-ua-verification/