I Logged Every AI Crawler for 34 Days. ChatGPT Outreads Googlebot

In mid-July, my Google clicks in my home market (Israel) dropped by almost half. Buyer-intent queries that used to bring steady leads just evaporated from Search Console. While I was staring at GSC dashboards trying to figure out what broke, I finally did the thing I should have done months earlier: I stopped looking at dashboards and started reading raw server logs.

What I found there was a parallel universe. Google Search was sending me less than ever — but AI systems were reading my site constantly. Not "someday this will matter" constantly. Right-now constantly: an AI assistant was fetching a page of mine roughly every 26 minutes, around the clock, because a real human had just asked it a question.

So I built a small log analyzer and let it run. Here's what 34 days of complete Caddy logs from a small business site (about 70 real human visitors a day) actually look like.

All counts are HTTP 200 responses only (more on why below), over 34 days:

Three things in that table genuinely surprised me.

ChatGPT-User outreads Googlebot. 40.3 fetches a day versus 35.8. This isn't a crawler building an index for later — ChatGPT-User

is the user-agent OpenAI sends when a human is mid-conversation and ChatGPT decides to pull a live page to answer them. On my site, that now happens more often than Googlebot visits. For a tiny business site in a niche market, I did not expect that.

Bing crawls 4.4x harder than Google. 158 requests a day versus 36. Nobody optimizes for Bing. Bing's referral traffic is a rounding error. But Seer Interactive found that 87% of SearchGPT's citations match Bing's top organic results (versus 56% for Google) — Bing's index is a structural input to what ChatGPT can retrieve and cite. Suddenly that 4.4:1 ratio reads very differently: the "irrelevant" search engine is the supply chain for the assistant that's actually fetching my pages 40 times a day.

Live retrieval is the biggest AI bucket that matters. Grouping by meaning rather than by bot name:

🔴 Live user retrieval (a human asked right now)   1,907  (~56/day)
🔎 AI-search indexing (OAI-SearchBot, Perplexity,
   bingbot, Applebot)                               6,096  (~177/day)
📚 Training crawls (GPTBot, ClaudeBot, Amazonbot)     897  (~26/day)
🌐 Classic Google                                   1,233  (~36/day)

1,907 live retrievals in 34 days. Each one of those is a moment where a real person asked an AI assistant something, and the assistant chose my site as a source. That's the leading indicator for citations — and it's the number I now track weekly.

Per-bot top pages turned out to be more interesting than the totals:

/signin

and /forgot-password

. Twelve visits to my login page. Enjoy the training data, I guess.The analyzer is ~60 lines of Python that runs over Caddy's JSON access logs. The core is just user-agent patterns with one ordering rule — the *-User

patterns must be matched before the generic bot patterns, because the loop stops at first match:

BOTS = {
    "ChatGPT-User":    re.compile(r"ChatGPT-User", re.I),   # live retrieval
    "Claude-User":     re.compile(r"Claude-User", re.I),    # live retrieval
    "Perplexity-User": re.compile(r"Perplexity-User", re.I),
    "OAI-SearchBot":   re.compile(r"OAI-SearchBot", re.I),
    "GPTBot":          re.compile(r"GPTBot", re.I),          # training
    "PerplexityBot":   re.compile(r"Perplexity", re.I),
}

The trap: count only HTTP 200s. My first version counted every request, and the numbers were inflated with garbage — security scanners spoof OpenAI user-agents (a scanner claiming to be GPTBot while probing for /wp-admin

on a site that doesn't run WordPress) and land on 404s. Filtering to 200-only removed the impostors and left actual content fetches. If you run this on your own logs and skip that filter, your "AI traffic" will be part vulnerability scan.

None of these 8,900 AI-related requests appear in GA4. No JavaScript execution, no consent banner click, no pageview event. And it's worse than that: on my site, GA4 undercounts humans by roughly 9x too — only ~11% of visitors accept the cookie banner (I measured this two independent ways over 11 days: ~780 real browser visitors in the logs and widget-loads, 89 in GA4).

So the fastest-growing reader of my site — AI assistants answering buyer questions in real time — is completely invisible unless you read server logs. If your hosting gives you raw access logs, that's where this story lives. If it doesn't, you can't see it at all, and you might be making content decisions while blind to a quarter of your real readership.

*-User

agents weeklyNow I want your denominator. Pull one month of access logs and count bingbot

vs Googlebot

hits — my ratio is 4.4:1. If you run a small site, especially a non-English one, I'd genuinely like to know whether that ratio holds outside my niche: drop your two numbers (and your site's language) in the comments.