{"slug": "ai-crawler-user-agents-are-self-reported-468-real-fetches-991-fake-ones", "title": "AI crawler user agents are self-reported: 468 real fetches, 991 fake ones", "summary": "A developer found that most AI crawler traffic on their site is fake, with only 468 of 1,459 requests actually fetching articles. Using Cloudflare's verifiedBotCategory, they showed that only 13% of GPTBot claims were verified, while meta-externalagent and Applebot were 100% verified. The developer provides a method to distinguish real AI crawlers from impersonators using free Cloudflare tools.", "body_md": "I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up.\n\nThen I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for `.env`\n\nand friends: 991. Of everything calling itself GPTBot, Cloudflare could verify 13% as OpenAI.\n\nHere is how to separate the impersonators on a Cloudflare free plan, and what the numbers looked like.\n\nWriting `GPTBot/1.2`\n\ninto a header costs nothing, so a table grouped by UA is a list of claims.\n\nBehind Cloudflare there are two things to check those claims against:\n\n`verifiedBotCategory`\n\nWhat survives both filters is \"verified bots fetching real pages\", and that is the only number worth reporting.\n\nAdd `userAgent`\n\nand `verifiedBotCategory`\n\nto the dimensions of `httpRequestsAdaptiveGroups`\n\n. `clientAsn`\n\nand `botClass`\n\nrequire a paid plan; `verifiedBotCategory`\n\ndoes not.\n\n```\nQUERY_BY_UA_VERIFIED = \"\"\"\nquery ($zoneTag: String!, $since: Time!, $until: Time!) {\n  viewer {\n    zones(filter: { zoneTag: $zoneTag }) {\n      httpRequestsAdaptiveGroups(\n        limit: 5000\n        filter: { datetime_geq: $since, datetime_leq: $until }\n        orderBy: [count_DESC]\n      ) {\n        count\n        dimensions { userAgent verifiedBotCategory }\n      }\n    }\n  }\n}\n\"\"\"\n```\n\nTwo constraints to plan around:\n\n``` python\ndef fetch_rows(token, query, zone_tag, since, until):\n    # split the range into one-day windows and concatenate the rows\n    all_rows, cursor = [], since\n    one_day = dt.timedelta(days=1)\n    while cursor < until:\n        win_end = min(cursor + one_day, until)\n        variables = {\n            \"zoneTag\": zone_tag,\n            \"since\": cursor.isoformat() + \"Z\",\n            \"until\": win_end.isoformat() + \"Z\",\n        }\n        try:\n            all_rows.extend(rows_from(gql(token, query, variables, exit_on_error=False)))\n        except RuntimeError as e:\n            print(f\"    [skipped] {cursor.date()}-{win_end.date()}: {e}\")   # past retention\n        cursor = win_end\n    return all_rows\n```\n\nThat single extra dimension is enough to split claim from reality. Eight days:\n\n| Claimed UA | Verified | Unverified | Verified share |\n|---|---|---|---|\n| ChatGPT-User | 211 (AI Assistant) | 336 | 39% |\n| Amazonbot | 135 (AI Crawler) | 489 | 22% |\n| ClaudeBot | 133 (AI Crawler) | 126 | 51% |\n| OAI-SearchBot | 61 (Search Engine Crawler) | 132 | 32% |\n| GPTBot | 19 (AI Crawler) | 123 | 13% |\n| meta-externalagent | 209 (AI Crawler) | 0 | 100% |\n| Applebot | 56 (AI Search) | 0 | 100% |\n| PerplexityBot | 0 | 144 | 0% |\n| Perplexity-User | 0 | 311 | 0% |\n\nOf 547 requests presenting as `ChatGPT-User`\n\n, 211 came from an address that traced back to OpenAI.\n\nOnly two agents came through clean — `meta-externalagent`\n\nand `Applebot`\n\n, 100% verified with zero impersonation. Those are the only rows whose claimed totals are usable as-is. All 455 Perplexity-branded requests were unverified.\n\n123 requests claimed `Google-Extended`\n\n. Verified share 0%, and 70 of them hit credential-scanning paths.\n\nNo inference required. Google's [crawler documentation](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) states that `Google-Extended`\n\nhas no separate HTTP request user agent string: crawling happens under the existing Google user agents, and the token exists purely to be addressed in robots.txt for AI-training control.\n\nSo every request presenting that UA is, by definition, not Google. The same trick works for any operator that publishes IP ranges — OpenAI ships [gptbot.json](https://openai.com/gptbot.json).\n\nVerification alone isn't enough: a verified bot fetching robots.txt has read nothing.\n\n```\nSCAN_PATTERNS = (\n    \"wp-\", \".env\", \".git\", \".aws\", \".svn\", \".ssh\", \"secrets\", \"credentials\",\n    \"config.json\", \"service_account\", \"actuator\", \"api/auth\", \"phpinfo\",\n    \".bak\", \".yml\", \".yaml\", \".php\", \".sql\", \"id_rsa\", \".npmrc\", \".htpasswd\",\n)\nOPS_PREFIXES = (\"/robots.txt\", \"/sitemap\", \"/llms.txt\", \"/favicon\", \"/rss\", \"/feed\", \"/.well-known/\")\nASSET_PREFIXES = (\"/_astro/\", \"/images/\", \"/assets/\", \"/fonts/\", \"/cdn-cgi/\", \"/_image\")\n\ndef classify_path(path, sitemap_paths):\n    # content = a real page was consumed / ops = crawl bookkeeping\n    # asset = static file / scan = credential probing / other = path does not exist\n    if not path:\n        return \"other\"\n    low = path.lower()\n    if any(k in low for k in SCAN_PATTERNS):\n        return \"scan\"\n    if low.startswith(OPS_PREFIXES):\n        return \"ops\"\n    if low.startswith(ASSET_PREFIXES):\n        return \"asset\"\n    if not sitemap_paths:\n        return \"unknown\"   # cannot assert existence, so cannot call it content\n    return \"content\" if (path.rstrip(\"/\") or \"/\") in sitemap_paths else \"other\"\n```\n\nUsing the sitemap as the source of truth for existence is the part that holds up. Deciding from the response status looks easier, but redirects and paths that answer 200 without being real pages both leak in. The set of paths you declared public is a cleaner definition of \"a page of mine\".\n\nKeep the `unknown`\n\nbranch too. Fold it into `content`\n\nand your numbers spike on any day the sitemap fetch fails, with nothing in the output to explain it.\n\n| Claimed UA | content | ops | scan | total | verified |\n|---|---|---|---|---|---|\n| ChatGPT-User | 216 | 0 | 181 | 547 | 39% |\n| meta-externalagent | 93 | 9 | 0 | 209 | 100% |\n| Amazonbot | 85 | 1 | 284 | 624 | 22% |\n| Applebot | 27 | 9 | 0 | 56 | 100% |\n| OAI-SearchBot | 17 | 43 | 74 | 193 | 32% |\n| PerplexityBot | 13 | 8 | 67 | 144 | 0% |\n| GPTBot | 7 | 9 | 72 | 142 | 13% |\n| ClaudeBot | 6 | 129 | 70 | 259 | 51% |\n| Google-Extended | 0 | 0 | 70 | 123 | 0% |\n| Perplexity-User | 0 | 0 | 173 | 311 | 0% |\n\n`content`\n\ntotals 468, `scan`\n\ntotals 991. By status code, 403s came to 957 against 705 served with 200.\n\nThe ClaudeBot row is worth pulling apart: of 133 verified requests, 6 fetched articles and 129 fetched robots.txt and similar. \"ClaudeBot sent 259 requests\" and \"six articles were read\" are the same data.\n\nKeep the **verified share as a column** in whatever you output. When a series' share collapses, that is your signal to stop reading it as a metric for that snapshot.\n\nThe week-over-week comparison (content 610 → 468 while scan went 357 → 991), why I cannot tell a genuine drop in interest apart from impersonation being reclassified, and the point where WAF blocks overtook served requests are all on Aulvem → [Aulvem | AI crawler user agents are self-reported: 468 real fetches, 991 fake ones](https://aulvem.com/blog/2026-08-22-ai-crawler-ua-verification/)", "url": "https://wpnews.pro/news/ai-crawler-user-agents-are-self-reported-468-real-fetches-991-fake-ones", "canonical_source": "https://dev.to/aulvem/ai-crawler-user-agents-are-self-reported-468-real-fetches-991-fake-ones-bgo", "published_at": "2026-08-22 03:31:26+00:00", "updated_at": "2026-08-22 04:14:28.002947+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools"], "entities": ["Cloudflare", "OpenAI", "GPTBot", "ChatGPT-User", "PerplexityBot", "Google-Extended", "Applebot", "meta-externalagent"], "alternates": {"html": "https://wpnews.pro/news/ai-crawler-user-agents-are-self-reported-468-real-fetches-991-fake-ones", "markdown": "https://wpnews.pro/news/ai-crawler-user-agents-are-self-reported-468-real-fetches-991-fake-ones.md", "text": "https://wpnews.pro/news/ai-crawler-user-agents-are-self-reported-468-real-fetches-991-fake-ones.txt", "jsonld": "https://wpnews.pro/news/ai-crawler-user-agents-are-self-reported-468-real-fetches-991-fake-ones.jsonld"}}