{"slug": "i-logged-every-ai-crawler-for-34-days-chatgpt-outreads-googlebot", "title": "I Logged Every AI Crawler for 34 Days. ChatGPT Outreads Googlebot", "summary": "A developer who logged AI crawler traffic on a small business site for 34 days found that ChatGPT's live retrieval bot (ChatGPT-User) outpaces Googlebot, fetching pages 40.3 times per day versus 35.8. The developer also discovered Bing crawls 4.4 times more than Google, and that live user retrieval—when an AI assistant pulls a page to answer a human query—is the largest meaningful AI traffic bucket, totaling 1,907 requests. The findings suggest that AI assistants are becoming a significant source of referral traffic, with Bing's index serving as a supply chain for ChatGPT's citations.", "body_md": "In mid-July, my Google clicks in my home market (Israel) dropped by almost half. Buyer-intent queries that used to bring steady leads just evaporated from Search Console. While I was staring at GSC dashboards trying to figure out what broke, I finally did the thing I should have done months earlier: I stopped looking at dashboards and started reading raw server logs.\n\nWhat I found there was a parallel universe. Google Search was sending me less than ever — but AI systems were reading my site *constantly*. Not \"someday this will matter\" constantly. Right-now constantly: an AI assistant was fetching a page of mine roughly every 26 minutes, around the clock, because a real human had just asked it a question.\n\nSo I built a small log analyzer and let it run. Here's what 34 days of complete Caddy logs from a small business site (about 70 real human visitors a day) actually look like.\n\nAll counts are HTTP 200 responses only (more on why below), over 34 days:\n\n| Bot | Requests | Per day | What it is |\n|---|---|---|---|\n| bingbot | 5,444 | 158.2 | Bing's index — which feeds ChatGPT |\nChatGPT-User |\n1,388 |\n40.3 |\nLive fetch while a human asks ChatGPT |\n| Googlebot | 1,233 | 35.8 | Classic Google crawl |\n| GPTBot | 547 | 15.9 | OpenAI training crawler |\n| Claude-User | 519 | 15.1 | Live fetch while a human asks Claude |\n| OAI-SearchBot | 281 | 8.2 | ChatGPT search indexing |\n| Applebot | 268 | 7.8 | Apple (Siri / Apple Intelligence) |\n| ClaudeBot | 214 | 6.2 | Anthropic training crawler |\n| Amazonbot | 136 | 4.0 | Amazon (Alexa & co.) |\n| PerplexityBot | 103 | 3.0 | Perplexity indexing |\n\nThree things in that table genuinely surprised me.\n\n**ChatGPT-User outreads Googlebot.** 40.3 fetches a day versus 35.8. This isn't a crawler building an index for later — `ChatGPT-User`\n\nis the user-agent OpenAI sends when a human is *mid-conversation* and ChatGPT decides to pull a live page to answer them. On my site, that now happens more often than Googlebot visits. For a tiny business site in a niche market, I did not expect that.\n\n**Bing crawls 4.4x harder than Google.** 158 requests a day versus 36. Nobody optimizes for Bing. Bing's referral traffic is a rounding error. But [Seer Interactive found that 87% of SearchGPT's citations match Bing's top organic results](https://www.seerinteractive.com/insights/87-percent-of-searchgpt-citations-match-bings-top-results) (versus 56% for Google) — Bing's index is a structural input to what ChatGPT can retrieve and cite. Suddenly that 4.4:1 ratio reads very differently: the \"irrelevant\" search engine is the supply chain for the assistant that's actually fetching my pages 40 times a day.\n\n**Live retrieval is the biggest AI bucket that matters.** Grouping by *meaning* rather than by bot name:\n\n```\n🔴 Live user retrieval (a human asked right now)   1,907  (~56/day)\n🔎 AI-search indexing (OAI-SearchBot, Perplexity,\n   bingbot, Applebot)                               6,096  (~177/day)\n📚 Training crawls (GPTBot, ClaudeBot, Amazonbot)     897  (~26/day)\n🌐 Classic Google                                   1,233  (~36/day)\n```\n\n1,907 live retrievals in 34 days. Each one of those is a moment where a real person asked an AI assistant something, and the assistant chose *my* site as a source. That's the leading indicator for citations — and it's the number I now track weekly.\n\nPer-bot top pages turned out to be more interesting than the totals:\n\n`/signin`\n\nand `/forgot-password`\n\n. Twelve visits to my login page. Enjoy the training data, I guess.The analyzer is ~60 lines of Python that runs over Caddy's JSON access logs. The core is just user-agent patterns with one ordering rule — the `*-User`\n\npatterns must be matched *before* the generic bot patterns, because the loop stops at first match:\n\n```\nBOTS = {\n    \"ChatGPT-User\":    re.compile(r\"ChatGPT-User\", re.I),   # live retrieval\n    \"Claude-User\":     re.compile(r\"Claude-User\", re.I),    # live retrieval\n    \"Perplexity-User\": re.compile(r\"Perplexity-User\", re.I),\n    \"OAI-SearchBot\":   re.compile(r\"OAI-SearchBot\", re.I),\n    \"GPTBot\":          re.compile(r\"GPTBot\", re.I),          # training\n    \"PerplexityBot\":   re.compile(r\"Perplexity\", re.I),\n    # ... bingbot, Googlebot, ClaudeBot, Applebot, Amazonbot\n}\n```\n\nThe trap: **count only HTTP 200s.** My first version counted every request, and the numbers were inflated with garbage — security scanners spoof OpenAI user-agents (a scanner claiming to be GPTBot while probing for `/wp-admin`\n\non a site that doesn't run WordPress) and land on 404s. Filtering to 200-only removed the impostors and left actual content fetches. If you run this on your own logs and skip that filter, your \"AI traffic\" will be part vulnerability scan.\n\nNone of these 8,900 AI-related requests appear in GA4. No JavaScript execution, no consent banner click, no pageview event. And it's worse than that: on my site, GA4 undercounts *humans* by roughly 9x too — only ~11% of visitors accept the cookie banner (I measured this two independent ways over 11 days: ~780 real browser visitors in the logs and widget-loads, 89 in GA4).\n\nSo the fastest-growing reader of my site — AI assistants answering buyer questions in real time — is completely invisible unless you read server logs. If your hosting gives you raw access logs, that's where this story lives. If it doesn't, you can't see it at all, and you might be making content decisions while blind to a quarter of your real readership.\n\n`*-User`\n\nagents weekly**Now I want your denominator.** Pull one month of access logs and count `bingbot`\n\nvs `Googlebot`\n\nhits — my ratio is 4.4:1. If you run a small site, especially a non-English one, I'd genuinely like to know whether that ratio holds outside my niche: drop your two numbers (and your site's language) in the comments.", "url": "https://wpnews.pro/news/i-logged-every-ai-crawler-for-34-days-chatgpt-outreads-googlebot", "canonical_source": "https://dev.to/achiya-automation/i-logged-every-ai-crawler-for-34-days-chatgpt-outreads-googlebot-369o", "published_at": "2026-08-16 09:40:07+00:00", "updated_at": "2026-08-16 10:12:03.573490+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-infrastructure", "developer-tools"], "entities": ["OpenAI", "Google", "Bing", "ChatGPT", "Claude", "Perplexity", "Apple", "Amazon"], "alternates": {"html": "https://wpnews.pro/news/i-logged-every-ai-crawler-for-34-days-chatgpt-outreads-googlebot", "markdown": "https://wpnews.pro/news/i-logged-every-ai-crawler-for-34-days-chatgpt-outreads-googlebot.md", "text": "https://wpnews.pro/news/i-logged-every-ai-crawler-for-34-days-chatgpt-outreads-googlebot.txt", "jsonld": "https://wpnews.pro/news/i-logged-every-ai-crawler-for-34-days-chatgpt-outreads-googlebot.jsonld"}}