I launched a site focused on Generative Engine Optimization (GEO) about 2 months ago. Instead of just tracking Google rankings, I set up Cloudflare D1 logging to track every AI crawler visit.
Here's what 30 days of data shows:
Crawler breakdown:
Key insights:
Bing is the new Google for AI. Bingbot volume dwarfs everything else. If you're optimizing for GEO, Bing Webmaster Tools is your best free analytics source.
ClaudeBot is remarkably consistent. Same crawl pattern every day, suggesting scheduled indexing rather than on-demand. If you publish content, expect 24-48hr lag before Claude "sees" it.
Bytespider (ByteDance) is aggressive. 500+ hits/day on a small site. They're clearly doing massive data collection. If you want visibility in Doubao/DeepSeek, your content needs to be crawlable.
External referrers matter more than I expected. A single Reddit post drove 8 AI crawler visits from a domain I'd never heard of (sahammurah.com). External mentions = AI discovery signals.
What I'm doing differently now:
Anyone else tracking AI crawlers? What patterns are you seeing?