{"slug": "98-5-of-our-googlebot-traffic-was-not-google", "title": "98.5% of our \"Googlebot\" traffic was not Google", "summary": "An operator of an MCP directory analyzed 22 hours of nginx logs from anclum.com and found that 6,031 of 6,121 requests claiming to be Googlebot — 98.5% — originated outside Google's published IP ranges, coming instead from scanners hunting for exposed secrets like /.env and /.git/config. Verifying against Google's crawler ranges showed only 258 genuine Google requests in the window, with a 404 rate of 2.3% rather than the 45% suggested by User-Agent matching, a discrepancy the team nearly acted on by redesigning parts of the site.", "body_md": "*Log analysis from a 22-hour window on anclum.com, November 7, 2026.*\n\nIf you measure how often Google visits your site, stop checking the User-Agent string. It is lying to you.\n\nWe run an MCP directory. Like everyone else, we watch how much of it Google actually crawls — that number decides whether anything we publish is ever seen. So we pulled 22 hours of nginx access logs and counted the requests that claim to be Googlebot.\n\n**6,121 requests said they were Googlebot.**\n\nThen we checked the source IP of every one of them against Google's published crawler ranges.\n\n**90 actually were. The other 6,031 — 98.5% of the 6,121 — came from outside Google's published IP ranges.** Not Google, not Bing, not any crawler that matters to us. They came from scanners, and they were wearing Google's name.\n\nThe 90 and the 258 are two different sets, and mixing them up is how this analysis goes wrong. 258 is the total number of requests Google's own IP ranges sent us in the same window — 168 of them with a User-Agent that does not say \"Googlebot\" at all. 90 is the overlap: requests that both claimed to be Googlebot and came from Google.\n\nHere is what the fake Googlebot was asking for:\n\n```\n/.env   /.env.local   /.env.production   /.env.bak\n/.git/config   /.git/head   /wp-admin/install.php?step=1\n/config.json   /api/.env   /.well-known/oauth-protected-resource\n```\n\nEvery one of those is an attempt to find a leaked secret or an unauthenticated admin panel. It is background radiation on the public internet — every site gets it, every day.\n\nThe damage is not the scanning. **The damage is what it does to your numbers.**\n\nIf you count bot traffic by User-Agent, your log says \"Google visited 6,121 times today.\" You feel good. You conclude your crawl budget is healthy. You might even conclude Google likes you.\n\nThe truth was 258 requests from Google's ranges — and the quality of what Google actually got looked completely different from what the fake traffic implied. Every share below is out of those 258:\n\n| Status | Requests (of 258) | Share | \n|---|---|---|\n| 200 OK | 200 | **77.5%** | \n| 410 Gone | 38 | 14.7% | \n| 301 redirect | 13 | 5.0% | \n| 404 | 6 | **2.3%** | \n| 302 redirect | 1 | 0.4% | \n\n**We had spent part of the day believing that 45% of those 6,121 requests were hitting dead links.** That number came from a User-Agent match. It was wrong by more than an order of magnitude. Of the 258 Google-range requests, the 404 rate was 2.3%.\n\nWe had been about to redesign a chunk of the site to fix a problem that did not exist — and to ignore the one that did: **Google's ranges sent us only 258 requests in 22 hours.**\n\nWe first wrote the analysis with a `grep 'Googlebot'` and a `uniq -c`. It produced a beautiful, tidy, completely wrong table.\n\nThe tell was in the data itself, and we nearly missed it: the \"dead links\" that accounted for 45% of the 6,121 UA-matched requests were things like `/.env.save` and `/wp-admin/install.php`. **Googlebot does not hunt for your `.env` file.** Apache and Nginx logs are full of this noise from a long list of botnets, and a large fraction of them now spoof the Googlebot UA — including ancient Android strings, because that is what real Googlebot used to look like.\n\nIf your 404 report has `/wp-admin` in it, you are not looking at a search engine. You are looking at an attacker who read a blog post about SEO.\n\nThree ways, in increasing order of correctness:\n\n`_cloud-netblocks.googleusercontent.com` (spf-style TXT records) and as `googlebot.json` from Google's developer docs. Match your log IPs against those.`dig -x <ip>` should return something ending in `googlebot.com` (or `google.com` for the other Google crawlers). Then forward-resolve that name and confirm it comes back to the same IP — otherwise you are being fooled by a spoofed PTR record.\nFor our first pass we used a hardcoded list of Google's crawler prefixes (`66.249.`, `64.233.`, `74.125.`, `209.85.`, `72.14.`, `216.239.`). That is an approximation, and it can both over- and under-count. The published ranges plus reverse DNS are the correct method — we now use them for the numbers we put on paper.\n\n258 requests from Google's ranges over 22 hours, touching **233 distinct URLs**. And the distribution was more interesting than the total. Every percentage below is a share of those 258:\n\n`/en/t/mcp/p/62/`, `/zh/t/from-github/az/p/15/` and friends. These are pure navigation: the same list of tools, sliced into pages. They are not in our sitemap and never have been, but they are linked from every list page, so Google keeps coming through the door.`/ny/contract/...` pages that no longer exist, and 38 of the 258 responses overall were `410 Gone` — Google confirming that content which used to exist does not. That is the correct signal and it decays on its own. Blocking those URLs in `robots.txt` at this point would be actively harmful: a blocked page cannot be seen to be gone.\n**65 requests in 22 hours to pagination.** That is the single largest waste in an otherwise healthy crawl, and it is fixable: `noindex, follow` on pagination templates. The `follow` matters, so the links still pass through.\n\nBut here is the part we had to admit to ourselves: **moving those 65 requests does not create more crawling.** The ceiling is what Google is willing to spend on us, and that ceiling is set by how much other sites point at us — not by how tidy our internal links are. **Optimizing the split of a small budget is still a small budget.**\n\n`.env` files.`wp-admin`, it is not Google.\nThe method above is reproducible from any nginx access log. If your numbers look alarming, check who you are actually counting before you fix anything.\n\n*Window: 2026-11-07 00:24:03–22:11:59 (Asia/Shanghai), nginx access log, 9.8 MiB. Real-crawler classification by Google's published crawler IP prefixes, with the caveat noted above. Source: anclum.com access log `access.log-20261108.gz`.*", "url": "https://wpnews.pro/news/98-5-of-our-googlebot-traffic-was-not-google", "canonical_source": "https://dev.to/anclum/985-of-our-googlebot-traffic-was-not-google-2k2j", "published_at": "2026-11-10 10:17:11+00:00", "updated_at": "2026-11-10 10:22:19.923062+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "developer-tools"], "entities": ["Google", "Googlebot", "anclum.com", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/98-5-of-our-googlebot-traffic-was-not-google", "markdown": "https://wpnews.pro/news/98-5-of-our-googlebot-traffic-was-not-google.md", "text": "https://wpnews.pro/news/98-5-of-our-googlebot-traffic-was-not-google.txt", "jsonld": "https://wpnews.pro/news/98-5-of-our-googlebot-traffic-was-not-google.jsonld"}}