{"slug": "ai-crawlers-now-match-googlebot-in-traffic-to-our-site", "title": "AI crawlers now match Googlebot in traffic to our site", "summary": "AI crawlers now match Googlebot in traffic to a website that hit the front page of Hacker News, with both categories accounting for 35% of 240,060 bot visits over 30 days. AmazonBot was the most aggressive AI crawler, while ChatGPT-User and ClaudeBot also ranked high. The data shows AI bots are now as prevalent as traditional search engine crawlers.", "body_md": "# AI crawlers now match Googlebot in traffic to our site\n\nWhat traffic & bots flow to your site after you make it to the front page of Hacker News? Do AI bots crawl as much as Google? Which is the most aggressive bot? Which is the least and thriftiest?\n\nA recent post hit the front page on hacker news. [Google just made you a search quality rater.You won't get paid. ](/blog/google-just-made-you-a-search-quality-rater-you-won-t-get-paid)\n\nThis was something I wished to know so I built a little bot detector on this site. Of course using Claude, the bot.\n\nThe code basically measured the traffic over 30 days and I built a simple analyser within the code to output the following data.\n\n**Across 240,060 visits from 24 distinct bots, AI crawlers (35.0% of traffic) tied search engines (35.0%) for the top slot.**\n\nIf you care about this industry then that's the headline. The rest is nerding in data.\n\n## The setup\n\nWe log every visit from a recognised bot user-agent into a single table. The fields are bot name, category, path, day, hit count. **240,060 hits, 24 bots, 5,074 distinct URLs touched.**\n\nWe bucket bots into five categories: **search** (Google, Bing, DuckDuckGo, etc.), **ai** (GPTBot, ClaudeBot, PerplexityBot, AmazonBot, etc.), **seo** (Ahrefs, Semrush, Majestic, Screaming Frog), **social** (LinkedIn, Facebook, Twitter, Pinterest), and **archive** (Internet Archive).\n\n## Category split\n\nAI crawlers tied search engines at 35.0% each.\n\nCategory | Hits | Share |\n|---|---|---|\nAI crawlers | 84,080 | 35.0% |\nSearch engines | 83,980 | 35.0% |\nSEO tools | 50,820 | 21.2% |\nSocial previews | 20,700 | 8.6% |\nArchive | 480 | 0.2% |\n\n## Most aggressive bots\n\nHits per day, averaged over the period.\n\nBot | Category | Hits/day | Total hits | Unique paths |\n|---|---|---|---|---|\nGooglebot | search | 1,228 | 36,840 | 1,474 |\nBingbot | search | 1,187 | 35,610 | 1,785 |\nAmazonBot | ai | 1,101 | 33,040 | 1,442 |\nMajesticBot | seo | 1,062 | 31,860 | 2,026 |\nChatGPT-User | ai | 812 | 24,350 | 239 |\nClaudeBot | ai | 581 | 17,430 | 647 |\nAhrefsBot | seo | 485 | 14,550 | 1,279 |\nLinkedInBot | social | 478 | 14,340 | 15 |\n\n## Most polite bots\n\nHits per path. Lower = visit and leave.\n\nBot | Category | Hits/path | Unique paths |\n|---|---|---|---|\nAhrefsBot | seo | 11.4 | 1,279 |\nApplebot | search | 11.9 | 575 |\nMajesticBot | seo | 15.7 | 2,026 |\nSemrushBot | seo | 16.2 | 265 |\nBingbot | search | 20.0 | 1,785 |\nAmazonBot | ai | 22.9 | 1,442 |\nGooglebot | search | 25.0 | 1,474 |\nClaudeBot | ai | 26.9 | 647 |\n\n## Most obsessive bots\n\nRe-scrapers. Few URLs, hammered repeatedly.\n\nBot | Hits/path | Total hits | Distinct paths |\n|---|---|---|---|\nLinkedInBot | 956.0 | 14,340 | 15 |\nFacebookBot | 170.3 | 6,130 | 36 |\nChatGPT-User | 101.9 | 24,350 | 239 |\nInternet Archive | 16.0 | 480 | 30 |\n\n## Most thorough explorer\n\nDistinct URLs touched.\n\nBot | Distinct paths visited | Total hits |\n|---|---|---|\nMajesticBot | 2,026 | 31,860 |\nBingbot | 1,785 | 35,610 |\nGooglebot | 1,474 | 36,840 |\nAmazonBot | 1,442 | 33,040 |\nAhrefsBot | 1,279 | 14,550 |\nClaudeBot | 647 | 17,430 |\nApplebot | 575 | 6,840 |\n\n## Rare visitors\n\nBots that barely show up across 30 days.\n\nBot | Total hits | Days active |\n|---|---|---|\nInternet Archive | 480 | 2 |\nBaiduspider | 480 | 9 |\nDuckDuckBot | 470 | 11 |\nTwitterbot | 160 | 4 |\nDotBot | 80 | 5 |\nPinterestbot | 70 | 4 |\nGoogle-Extended (Gemini) | 60 | 1 |\nScreaming Frog | 50 | 5 |\n\n### Methodology\n\n**Source:**`nexus_bot_visits`\n\ntable on mojodojo.io's production database. One row per (bot, path, day) with a hit counter.**Bot detection:** User-Agent substring match against a curated list of 32 known bot patterns. Unknown / spoofed UAs are excluded.\n\nFiled under", "url": "https://wpnews.pro/news/ai-crawlers-now-match-googlebot-in-traffic-to-our-site", "canonical_source": "https://mojodojo.io/blog/ai-crawlers-match-googlebot-traffic", "published_at": "2026-06-17 16:42:43+00:00", "updated_at": "2026-06-17 16:52:54.882310+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-tools", "artificial-intelligence", "generative-ai"], "entities": ["Googlebot", "Bingbot", "AmazonBot", "ClaudeBot", "ChatGPT-User", "AhrefsBot", "LinkedInBot", "MajesticBot"], "alternates": {"html": "https://wpnews.pro/news/ai-crawlers-now-match-googlebot-in-traffic-to-our-site", "markdown": "https://wpnews.pro/news/ai-crawlers-now-match-googlebot-in-traffic-to-our-site.md", "text": "https://wpnews.pro/news/ai-crawlers-now-match-googlebot-in-traffic-to-our-site.txt", "jsonld": "https://wpnews.pro/news/ai-crawlers-now-match-googlebot-in-traffic-to-our-site.jsonld"}}