{"slug": "how-google-sees-your-site", "title": "How Google Sees Your Site", "summary": "A technical analysis of HAProxy logs using JA4 TLS fingerprints, user agents, and ASN data identified Google bots that are not listed in Google's own documentation, and the author notes that robots.txt is insufficient because Google admits some of its bots will not respect those rules. The examination covers Google traffic hitting a site, including the Google Other crawler used for AI training opt-out decisions, and surfaces malformed entries in the IP fields of the logs.", "body_md": "I keep stumbling over different Google bots in my logs. So today I'm looking at each and every Googly eye that takes a look at my site. Because when you stare at me, I'll stare back.\n\nYou have the right and the means to define what Google is allowed to do on your site, whether the Google traffic you face is extensive or you'd like to opt-out of AI training ([check Google Other](https://radar.cloudflare.com/bots/directory/google-other)). Sadly, using [robots.txt](https://en.wikipedia.org/wiki/Robots.txt) is not enough, Google admits that some of their bots [will not respect](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers#google-safety) the rules.\n\nPrevious articles were about [TLS fingerprinting JA3/JA4](/ja3ja4-tls-client-fingerprinting/) and how to [deploy it on HAProxy](/deploying-ja4/). In this article/[marimo](https://marimo.io/) notebook ([full source](https://gist.github.com/ArcHound/89d8158af8da45014745a3e5cf2d1688)) I'll use the HAProxy logs containing IP addresses (plus [ASNs](/asn-check/)), user agents and JA4 hashes to examine Google traffic. We'll see a couple of Google bots that are not mentioned in their docs.\n\nLet's get into it.\n\nThis article doubles as a [marimo](https://marimo.io/) notebook (of course, it's not modifiable and executable, that's a security nightmare to setup). Full [source here](https://gist.github.com/ArcHound/89d8158af8da45014745a3e5cf2d1688). So to kick things off, I'll do some arbitrary global imports. There are a couple dependencies like my own asn-check ([source](https://github.com/ArcHound/asn-check), [article](/asn-check/)) and [requests](https://pypi.org/project/requests/).\n\n``` python\nimport csv\nfrom ipaddress import ip_address, IPv4Network, IPv6Network, IPv4Address, IPv6Address\nfrom asn_check import ASNChecker\nfrom asn_check.ip_binary_tree import IPTree\nfrom typing import Union\nimport requests\nimport dataclasses\nfrom enum import Enum\nimport re\nfrom collections import defaultdict\n```\n\nWe need logs. First, I set up HAProxy (check out [the previous part](/deploying-ja4/)). Then I parsed the logs into something more sane, e.g. [like this](https://gist.github.com/ArcHound/cdecf272b3d07e402b168b0ffdf6c331). That \"format\" optimizes lookups, so I've forced it to CSV for easier processing. [Here's the CSV file.](/documents/1/stats.csv) Here's a sample of what we're working with:\n\n```\nwith open('haproxy_logs/stats.csv') as f:\n    cr = csv.DictReader(f)\n    data = [row for row in cr]\n\ndata[0]\n{\n\"ja4\":\n\"t13d591000_a33745022dd6_1f22a2ca17c4\"\n\"user_agent\":\n\"Uptime-Kuma/1.23.15\"\n\"ip\":\n\"161.97.71.61\"\n\"count\":\n\"157907\"\n}\n```\n\nGenerally, the signals to look for are IP addresses, User-Agents and the overall traffic. My new signal here is JA4, which I hope will give us some clarity on classifying these bots.\n\nAnother useful signal is ASN ([Autonomous System Number](<https://en.wikipedia.org/wiki/Autonomous_system_(Internet)>)). If IP is an address, then ASN is a city or a ZIP code. I have my [own lib](/asn-check/) to get these numbers, so let's fetch them. Initialization is done in a separate cell, it takes some time (~4min) to build the trees.\n\n```\nasn_checker = ASNChecker()\n```\n\nThe benefit is that consequent searches are fast[<sup>1</sup>](#footnote-1), so we can replay them as we'd like. And it's already paying off, since there's trash even in the IP fields!\n\n```\nfor r in data:\n    try:\n        asn = asn_checker.search(ip_address(r[\"ip\"]))\n        r[\"asn\"] = asn[\"asn\"]\n        r[\"as_name\"] = asn[\"name\"]\n        r[\"country_code\"] = asn[\"country_code\"]\n    except ValueError as e:\n        print(r)\n        print(e)\n{'ja4': 't13d250900_b78ed14e2fd0_e7c285222651', 'user_agent': '', 'ip': '} \"SSTP_DUPLEX_POST /sra_{BA195980-CD49-458b-9E23-C84EE0ADCD75', 'count': '11'}\n'} \"SSTP_DUPLEX_POST /sra_{BA195980-CD49-458b-9E23-C84EE0ADCD75' does not appear to be an IPv4 or IPv6 address\n{'ja4': '', 'user_agent': '', 'ip': '} \"GET /graphql?query=+{customerDownloadableProducts+{+items+{+date+download_url}}+', 'count': '1'}\n'} \"GET /graphql?query=+{customerDownloadableProducts+{+items+{+date+download_url}}+' does not appear to be an IPv4 or IPv6 address\n{'ja4': 't13d1412h2_e33ad33b3d25_6b314db333b6', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.7 Safari/537.36', 'ip': '2a06:98c0:3600:', 'count': '1'}\n'2a06:98c0:3600:' does not appear to be an IPv4 or IPv6 address\n{'ja4': 't13d1412h2_e33ad33b3d25_6b314db333b6', 'user_agent': \"Don't Hype Me RSS Reader/1.0\", 'ip': '2a06:98c0:3600:', 'count': '1330'}\n'2a06:98c0:3600:' does not appear to be an IPv4 or IPv6 address\n{'ja4': 't13d1412h2_e33ad33b3d25_6b314db333b6', 'user_agent': 'minifeed_net', 'ip': '2a06:98c0:3600:', 'count': '491'}\n'2a06:98c0:3600:' does not appear to be an IPv4 or IPv6 address\n```\n\nIn the log collection, I'm preferring the `X-Forwarded-For` header for the IP address, but that creates the opportunity to feed trash into this header in an attempt to break/exploit the parser of these data. Well, here I am, writing my parser around that.\n\nWe see the `2a06:98c0:3600:` entry a couple of times. That's an IPv6 prefix, not a full address, so the lib is complaining rightfully. It belongs to [Cloudflare](https://www.abuseipdb.com/check/2a06:98c0:3600::103) and it should be a reverse proxy. Seems that some Cloudflare customers are abusing this. I imagine you can order a reverse proxy to point it to a target you don't own. Then if you generate traffic to that proxy, the target gets hit. The key difference is that this IP is usually white-listed and therefore these attacks are hard to block. Internet is fun!\n\nLet's finish this stage of parsing by defining a log entry:\n\n```\n@dataclasses.dataclass(frozen=True)\nclass LogEntry:\n    ja4: str\n    user_agent: str\n    ip: Union[IPv4Address, IPv6Address]\n    count: int\n    asn: str # not all ASNs might be numbers and we are not using any number operations on them.\n    as_name: str\n    country_code: str\n```\n\nAnd let's also finish up the data clean-up by wrangling our data into this format. I employ some dirty tricks for brevity. The kwargs trick is especially useful with a dictionary -> dataclass transformation, but you have to conform to the format exactly.\n\n```\nlog_entries = list()\nfor entry in data:\n    try:\n        new_entry = dict(entry)\n        new_entry[\"ip\"] = IPv4Address(entry[\"ip\"]) if \":\" not in entry[\"ip\"] else IPv6Address(entry[\"ip\"])\n        new_entry[\"count\"] = int(entry[\"count\"])\n        log_entries.append(LogEntry(**new_entry))\n    except Exception as e:\n        print(f\"Malformed entry {entry}, exception {e}\")\nlog_entries[0]\nLogEntry(\n    ja4='t13d591000_a33745022dd6_1f22a2ca17c4', \n    user_agent='Uptime-Kuma/1.23.15', \n    ip=IPv4Address('161.97.71.61'), \n    count=157907, \n    asn='51167', \n    as_name='CONTABO Contabo GmbH', \n    country_code='DE'\n)\n```\n\nIn this section, let's map what we know about Google assets to identify traffic from Google.\n\nGoogle has [a lot of bots](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers). Let me manually extract the \"robots.txt\" versions of user-agents. That should be an immutable string that appears in all versions and flavors of that particular bot. Here:\n\n```\ngoogle_bot_ids = [\n    # https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers\n    \"Googlebot\",\n    \"Googlebot-Image\",\n    \"Googlebot-Video\",\n    \"Googlebot-News\",\n    \"Storebot-Google\",\n    \"Google-InspectionTool\",\n    \"GoogleOther\",\n    \"GoogleOther-Image\",\n    \"GoogleOther-Video\",\n    \"Google-CloudVertexBot\",\n    \"Google-Extended\",\n    # https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers\n    \"APIs-Google\",\n    \"AdsBot-Google\",\n    \"AdsBot-Google-Mobile\",\n    \"Mediapartners-Google\",\n    \"Google-Safety\",\n    \"DuplexWeb-Google\",\n    \"Google Favicon\",\n    \"AdsBot-Google-Mobile-Apps\",\n    \"googleweblight\",\n    # https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers\n    \"Google-CWS\",\n    \"FeedFetcher-Google\",\n    \"Google-GeminiNotebook\",\n    \"Google-NotebookLM\",\n    \"Google-Agent\",\n    \"GoogleMessages\",\n    \"Google-Pinpoint\",\n    \"GoogleProducer\",\n    \"Google-Read-Aloud\",\n    \"google-speakr\",\n    \"Google-Site-Verification\"\n]\nassert all([\"google\" in x.lower() for x in google_bot_ids])\n```\n\nNoticed something? Literally all of them have \"google\" in them in one way shape or form, so we can filter them out based on user-agents like so:\n\n```\ngoogle_ua_entries = [x for x in log_entries if 'google' in x.user_agent.lower()]\ngoogle_ja4s = {x.ja4 for x in google_ua_entries}\ngoogle_ja4_uas = {x:{y.user_agent for y in google_ua_entries if y.ja4==x} for x in google_ja4s}\n```\n\nAnd if you ask why didn't I use that nice list to check for the Google bots, I'll show you in a later section. Spoilers: there are more bots.\n\nAnother view we can try is to take a look at ASNs owned by Google and the traffic coming from those IPs:\n\n```\ngoogle_asn_entries = [x for x in log_entries if 'goog' in x.as_name.lower() or 'alphabet' in x.as_name.lower()]\ngoogle_asns = {(x.asn, x.as_name) for x in google_asn_entries}\ngoogle_asns\n{\n    ('396982', 'GOOGLE-CLOUD-PLATFORM - Google LLC'), \n    ('16591', 'GOOGLE-FIBER - Google Fiber Inc.'), \n    ('394089', 'GCP-ENTERPRISE-USER-TRAFFIC - Google LLC'), \n    ('15169', 'GOOGLE - Google LLC')\n}\n```\n\nNot much info is available online about the purpose of these ASNs. But let's try anyway.\n\n**ASN 16591: Google Fiber** [(IPInfo details)](https://ipinfo.io/AS16591)\n\nThis one is clear - it's [google's internet offering](https://gfiber.com/). If you ask me, Google has already way too much control over browsing, this is then the ultimate final step: your internet connection is fully owned by Google. Of course, they need an ASN for their fleet of routers. What's nice that this is an ISP type ASN that is treated differently than hosting type ASNs.\n\n**ASN 396982: Google Cloud Platform** [(IPInfo details)](https://ipinfo.io/AS396982)\n\nYou can rent a piece of Google in their Google Cloud Platform offering. It's a cloud hyperscaler, but you know this. I expect a lot of trash coming from this ASN as people might abuse the free tiers.\n\n**ASN 394089: GCP Enterprise User Traffic** [(IPInfo details)](https://ipinfo.io/AS394089)\n\nThis one I suspect is what Google can give you if you have assets in GCP plus a better contract. Otherwise I expect the same as in Google Cloud Platform ASN396982. As they say, if you have to ask, you can't afford it.\n\n**ASN 15169: Google** [(IPInfo details)](https://ipinfo.io/AS15169)\n\nThat leaves us with this general ASN. I can only presume that this is the traffic from all the rest of Google Internal services. It's explicitly not traffic from GCP workloads and from users hooked up to Google Fiber ISP. I make the conclusion that this is \"official Google traffic.\" I've found this [list of prefixes](https://www.gstatic.com/ipranges/cloud.json) by searching [in their docs](https://docs.cloud.google.com/vpc/docs/subnets#subnet-types).\n\nTo take a break from python, here's the breakdown after I ran a sophisticated bash analytics pipeline:\n\n``` bash\n$ wget -O- https://www.gstatic.com/ipranges/cloud.json 2>/dev/null | \\\njq | grep 'ipv4Prefix' | cut -f2 -d':' | grep -o '\"[^\"]*\"' | tr -d '\"' |\\\nsed 's?./.*$?9?' | asn-check |\\\ncut -f3 -d',' | sort | uniq -c | grep -v 'name'\n\n    909 GOOGLE-CLOUD-PLATFORM - Google LLC\n     81 GOOGLE - Google LLC\n      7 LEVEL3 - Level 3 Parent\n```\n\nIt's at this point when we can count the percentage of traffic coming from Google to see how it stacks against the rest of the internet. Note, that I have counts of requests in the LogEntry, now's the time to use them.\n\nI'll also exclude traffic from my monitoring tool (Uptime Kuma) as it would probably dominate the conversation.\n\n```\nno_monitor = [x for x in log_entries if x.user_agent != \"Uptime-Kuma/1.23.15\"]\ntotal_sum = sum([x.count for x in no_monitor])\ngoogle_sums = {f\"{x[0]}: {x[1]}\":sum([y.count for y in log_entries if y.asn==x[0]]) for x in google_asns}\npercentages = {x:f\"{google_sums[x]} -> {round(google_sums[x]*100/total_sum,2)}%\" for x in google_sums}\npercentages\nphp\n{\n  \"396982: GOOGLE-CLOUD-PLATFORM - Google LLC\": \"9617 -> 3.09%\",\n  \"16591: GOOGLE-FIBER - Google Fiber Inc.\": \"116 -> 0.04%\",\n  \"394089: GCP-ENTERPRISE-USER-TRAFFIC - Google LLC\": \"88 -> 0.03%\",\n  \"15169: GOOGLE - Google LLC\": \"1239 -> 0.4%\"\n}\n```\n\nNot gonna lie, I was hoping for a bigger number here so that I could have a great clickbait headline. But alas, seems that Google has its bots under control. The 3% at GCP ASN might very well be Google customers, not Google and still it isn't that much.\n\nNow let's take a look at who is trying to impersonate google:\n\n```\ngoogle_as_numbers = [x[0] for x in google_asns]\ngoogle_ua_non_google_asn = [x for x in log_entries if x.asn not in google_as_numbers and \"google\" in x.user_agent.lower()]\nimp_count = sum([x.count for x in google_ua_non_google_asn])\nf\"{imp_count} -> {round(imp_count*100/total_sum,2)}%\"\nphp\n608 -> 0.2%\n```\n\nAlso a nothing burger. I have attempted science today. Sometimes the dramatic story just isn't there and now I have to think of another clickbaity headline.\n\nNow let's take a look at user agents with \"google\" in them that also come from Google ASNs. That's the strongest signal we have for official Google traffic.\n\n```\ngoogle_ua_set = {x for x in google_ua_entries}\ngoogle_asn_set = {x for x in google_asn_entries}\nassert len(google_ua_set) - len(google_ua_entries) == 0 # based on log-collecting methods, this should be true\nassert len(google_asn_set) - len(google_asn_entries) == 0\ngoogle_entries = google_ua_set & google_asn_set\nf\"{len(google_ua_set)} entries have 'google' in them, {len(google_asn_set)} entries come from google ASNs, {len(google_entries)} have both.\"\n\"286 entries have 'google' in them, 4879 entries come from google ASNs, 148 have both.\"\n```\n\nLet's break it down by the user-agent:\n\n```\ngoogle_uas = {x.user_agent for x in google_entries}\ngoogle_uas_map = {x:[y for y in google_entries if y.user_agent==x] for x in google_uas}\ngoogle_uas_map\n```\n\nHere's the [full list attached](/documents/2/google_ua_by_ua.txt) - I'll be analyzing it shortly and it'd break the flow of the article if I included it directly. The order is, of course, arbitrary. Also note, that some of these bots come from GCP rather than pure Google LLC. While Google might use GCP ranges for official traffic, it's more likely one of GCP customers impersonating Google.\n\nBy far the most common bot, this is the one that brings your pages into the google search results ([docs](https://developers.google.com/search/docs/crawling-indexing/googlebot)). Comes in various different flavors (desktop and mobile), I guess it wants to see if there's the same content for various user agents.\n\nThe Image and Video variants are similar, but they feed the multi-media results into the respective sections of Google search.\n\nSo if you want your pages to be searchable by THE search engine, you'll have to allow these bots.\n\n**BEWARE!** These are also the most commonly impersonated bots on the internet. Since everyone wants to be searchable by Google, admins generally allow these user agents and the attackers will happily impersonate them. Take a look at these two entries:\n\n```\nLogEntry(ja4='t13d190900_9dc949149365_97f8aa674fd9', user_agent='Googlebot-Image/1.0', ip=IPv4Address('34.181.217.51'), count=1, asn='396982', as_name='GOOGLE-CLOUD-PLATFORM - Google LLC', country_code='US')\nLogEntry(ja4='t13d181300_e8a523a41297_43ade6aba3df', user_agent='Googlebot-Image/1.0', ip=IPv4Address('66.249.74.4'), count=23, asn='15169', as_name='GOOGLE - Google LLC', country_code='US')\n```\n\nOne comes from GCP, so it might be anyone. The other comes from Google LLC so it seems to be directly from Google. The JA4 hashes are also different, which is a hint that this is weird. We'll meet the `t13d190900_9dc949149365_97f8aa674fd9` hash again and I suspect this one is a spammer. So let's take a look at it:\n\nFor completeness, here's a quick breakdown of the listed bots:\n\nAlso in this category there's a user agent `BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0)`. I haven't found any WAP proxy service by google, but I found [one reference](https://user-agents.net/string/blackberry7520-4-0-0-profile-midp-2-0-configuration-cldc-1-1-up-browser-5-0-3-3-up-link-5-1-2-12-google-wap-proxy-1-0) for this user-agent.\n\nWell actually these all come from Google GCP ASN, so they might be impostors. And if you'd look at the JA4 hashes, this one `t13d190900_9dc949149365_97f8aa674fd9` is different from the usual `t13d181300_e8a523a41297_43ade6aba3df`. If I take even closer look, I see it in this batch:\n\n```\n... 35.245.141.163:47184 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Linux; Android 7.0; LGMS428) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.111 Mobile Safari/537.36|} \"GET /backups/db.sql HTTP/1.1\"\n... 35.245.141.163:47196 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36|} \"GET /tmp/backup.sql HTTP/1.1\"\n... 35.245.141.163:47176 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 MicroMessenger/7.0.5(0x17000523) NetType/4G Language/zh_CN|} \"GET /backup/dump.sql HTTP/1.1\"\n... 35.245.141.163:47210 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0|} \"GET /tmp/dump.sql HTTP/1.1\"\n... 35.245.141.163:47222 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13|} \"GET /exports/db.sql HTTP/1.1\"\n... 35.245.141.163:47188 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/4.0 (compatible; GoogleToolbar 4.0.1019.5266-big; Windows XP 5.1; MSIE 6.0.2900.2180)|} \"GET /backups/dump.sql HTTP/1.1\"\n... 35.245.141.163:47248 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Linux; Android 7.0; EVA-L09) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.111 Mobile Safari/537.36|} \"GET /config.js HTTP/1.1\"\n... 35.245.141.163:47232 ... {t13d190900_9dc949149365_97f8aa674fd9|Peach/1.01 (Ubuntu 8.04 LTS; U; en)|} \"GET /config.php HTTP/1.1\"\n... 35.245.141.163:47264 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; Linux x86_64; en-US; rv:2.0b2pre) Gecko/20100712 Minefield/4.0b2pre|} \"GET /config.yml HTTP/1.1\"\n... 35.245.141.163:47256 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36|} \"GET /config.json HTTP/1.1\"\n... 35.245.141.163:47276 ... {t13d190900_9dc949149365_97f8aa674fd9|Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 YaBrowser/17.3.0.1785 Yowser/2.5 Safari/537.36|} \"GET /config.yaml HTTP/1.1\"\n```\n\nThis is a textbook example of a random spray and pray vulnerability scan. So I conclude this is in fact not Google, but someone sending trash from GCP while rotating the user agents randomly.\n\nThere are a couple of Google AppsScript user agents, e.g. `Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +`` https://script.google.com``; id: UAEmdDd_1zdKXtniWEDkKiDKLGH6k2AjErCM)` that are related to the [Google Apps Script offering](https://developers.google.com/apps-script/reference/url-fetch).\n\nThat means these are most likely not directly Google initiated, but Google-user initiated. I appreciate the inclusion of the script ID, but I didn't find a way to report a misbehaving script, so that's a minus from me.\n\nGoogle has a concept of \"user initiated fetch\" (see [docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)). So in a sense these are user-initiated but coming from the Google infra. I've observed these:\n\nIn a sense, these are AI bots. But the stated intent is to gather context to execute agents and not to train new AI.\n\nSo there's this user agent `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.96 Mobile Safari/537.36 (compatible; Google-AMPHTML)` which can be identified by the `Google-AMPHTML` string. Apparently, this is a bot used to identify Accelerated Mobile Pages (AMP).\n\nThe issue is, I didn't find this one in the Google official documentation, nor did I find it in any google source when googling. Using reverse DNS on the [64.233.172.233](https://ipinfo.io/64.233.172.233?lookup_source=search-bar) IP I get `google-proxy-64-233-172-233.google.com` which also gives me nothing.\n\nNot cool google, not cool.\n\nWhile it is recognized in [the docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers#google-safety), there is only a general description of\n\nThe Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties.\n\nAnd I also love to see that:\n\nThe Google-Safety user agent ignores robots.txt rules.\n\nSo... if a link to my site somehow gets onto any Google property, they reserve the right to ignore my wishes for external crawlers to do what they need to do to ensure \"safety\" (whatever that means). A \"Magic The Gathering\" player in me recognizes the killer combo with the GoogleBot search spider. Yay.\n\nBy far my favourite user agent on this list: `Google`. It's even better that it comes from the official Google IP range and not GCP. Like what do you want to know you silly admin? It's Google, that'll have to do.\n\nSo I guess hello John Google, enjoy your stay.\n\nLooking at the reverse DNS for the [108.177.64.74](https://ipinfo.io/108.177.64.74?lookup_source=search-bar) address I get `rate-limited-proxy-108-177-64-74.google.com`. Surely, Google wouldn't re-route the traffic and anonymize the User Agent when it detects rate-limiting (which I have) to ignore the website wishes. No, Google would never.\n\nWell of course, the Other bot traffic. The bot traffic that is other. This other traffic is unlike any traffic that we've seen. There are apparently other use cases that are not covered by the previous bots and product portfolio. There are other things Google wants to do. Don't worry, every other thing ends well.\n\nThe thing is, this one is actually recognized in the official documentation. By far my least favourite because it's so opaque. And keep in mind, most of these come from Google LLC ASN and not the GCP ASN, so it's \"official\". I think if you're such a big company like Google and you're setting multiple random bots my way, you should have the courtesy to explain your reasons. And not just \"yea, we're looking at your site because we can, what are you going to do about it\"?\n\nSomehow, [Cloudflare assigned the AI CRAWLER category](https://radar.cloudflare.com/bots/directory/google-other) to this bot. Cloudflare claims that this bot gathers data for AI training purposes. I don't know why, but I also don't know how to disprove this so... bear that in mind.\n\nRecall there are these entries that come from Google ASNs and yet don't have \"google\" mentioned in their user agent. Let's break it down by ASN:\n\n```\nnon_google_google_asn_set = [x for x in google_asn_entries if x not in google_entries]\nsorted_non_google_google_asn = {x:[y for y in non_google_google_asn_set if y.asn==x[0]] for x in google_asns}\nsorted_non_google_google_asn\n```\n\nThe full list adds little to the article, but it can be found [here](/documents/3/google_ua_by_asn.txt) for those interested. Once again, let's break it down into parts.\n\nAs expected, most of the trash comes from GCP where anybody can run anything. By trash I mean things like:\n\n`xmlrpc.php` is my top 404),`/.env`, `/backup.sql` and others),\nMaybe the most fun info here is seeing Claude Code in the Enterprise GCP tier, [confirming their claims of partnership](https://www.anthropic.com/news/expanding-our-use-of-google-cloud-tpus-and-services). But ask yourself again, why would Claude Code, primarily a desktop application live in a cloud server?\n\nI plan to do a proper analysis of that traffic, but as I was digging around this I've hit Google so many times that I'm starting here and excluding all of the weird Google quirks from the main article. Sorry for now and TBD.\n\nNot much to see here, it's a small provider and it provides somewhat sane traffic.\n\nThere is one fun user-agent though:\n\n```\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0\n```\n\nAs per [this question](https://webmasters.stackexchange.com/questions/137914/spike-in-traffic-from-facebot-twitterbot-user-agent), this is a service of Apple that pre-fetches links for previews. I'd appreciate if Apple docs could confirm this, but it is what it is.\n\nThere is some weird traffic coming also from Google LLC. Recall, these shouldn't be Google customers, but Google itself as we've covered the other cases previously. Let's group it by user-agents:\n\n```\nuas_15169 = {x.user_agent for x in sorted_non_google_google_asn[(\"15169\", \"GOOGLE - Google LLC\")]}\nofficial_google_traffic_lol = {x:[y for y in sorted_non_google_google_asn[(\"15169\", \"GOOGLE - Google LLC\")] if y.user_agent==x] for x in uas_15169}\n{x: len(official_google_traffic_lol[x]) for x in official_google_traffic_lol}\n{\n  \"Chrome Privacy Preserving Prefetch Proxy\": 26,\n  \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36\": 84,\n  \"\": 3,\n  \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36\": 12,\n  \"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Mobile Safari/537.36\": 20,\n  \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36\": 1,\n  \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36\": 1,\n  \"Mozilla/5.0\": 2,\n  \"Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36\": 1,\n  \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36\": 1,\n  \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36\": 39,\n  \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36\": 44,\n  \"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Mobile Safari/537.36\": 5,\n  \"Mozilla/5.0 (iPhone; CPU iPhone OS 18_3_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3.1 Mobile/15E148 Safari/604.1\": 1,\n  \"Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36\": 1,\n  \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36\": 1,\n  \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36\": 16,\n  \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36\": 17,\n  \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj) Chrome/124.0.0.0 Safari/537.36\": 6,\n  \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36\": 3,\n  \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36\": 23,\n  \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36\": 1,\n  \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36\": 1,\n  \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36\": 7,\n  \"abuse.xmco.fr\": 2,\n  \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0 abuse.xmco.fr\": 2\n}\n```\n\nFor the first time in forever, I can call upon JA4 to group the rest of the entries:\n\n```\nja4s_15169 = {x.ja4 for x in sorted_non_google_google_asn[(\"15169\", \"GOOGLE - Google LLC\")]}\nofficial_google_traffic_ja4 = {x:[y for y in sorted_non_google_google_asn[(\"15169\", \"GOOGLE - Google LLC\")] if y.ja4==x] for x in ja4s_15169}\nuas_ja4s = {x:list({y.user_agent for y in official_google_traffic_ja4[x]}) for x in official_google_traffic_ja4}\nuas_ja4s_keys_sorted = sorted(list(uas_ja4s.keys()))\nfor _k in uas_ja4s_keys_sorted:\n    print(_k)\n    for _ua in uas_ja4s[_k]:\n        print(f\"    {_ua}\")\nt13d1514h2_8daaf6152771_827b515c4f52\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36\nt13d1516h2_8daaf6152771_02713d6af862\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36\nt13d1516h2_8daaf6152771_806a8c22fdea\n    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36\n    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36\n    Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Mobile Safari/537.36\nt13d1516h2_8daaf6152771_d8a2da3f94cd\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36\n    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36\n    Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36\n    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36\n    Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Mobile Safari/537.36\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36\n    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36\n    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36\n    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36\n    Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36\n    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36\n    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36\nt13d171000_5b57614c22b0_78e6aca7449b\n    Mozilla/5.0 (iPhone; CPU iPhone OS 18_3_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3.1 Mobile/15E148 Safari/604.1\nt13d181300_e8a523a41297_43ade6aba3df\n    Chrome Privacy Preserving Prefetch Proxy\nt13d190900_9dc949149365_e7c285222651\n\n    Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0 abuse.xmco.fr\n    abuse.xmco.fr\nt13d1909h2_9dc949149365_97f8aa674fd9\n    Mozilla/5.0\nt13d5212h1_b262b3658495_8e6e362c5eac\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj) Chrome/124.0.0.0 Safari/537.36\nt13d5213h1_b262b3658495_66863fb0a24c\n    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj) Chrome/124.0.0.0 Safari/537.36\n```\n\nWe can clearly see a couple of groups here. Even though the JA4 hashes are not matching exactly, the similarities are strong enough to predict some trends.\n\nFirst is the [Chrome privacy preserving prefetch proxy](https://developer.chrome.com/blog/private-prefetch-proxy) (CPPPP) that has a surprisingly unique JA4 hash of `t13d181300_e8a523a41297_43ade6aba3df`.\n\nThen we have the `t13d190900_9dc949149365_.*` group that seems to be linked to abuse.xmco.fr, a [French security scanner](https://www.xmco.fr/). While it's slightly surprising that this comes from the Google LLC ASN and not GCP ASN, the addresses are quite similar and I can imagine that this IP range was originally in the GCP. Or they did have some agreement with Google, IDK. It seems highly unlikely to me that Google would decide to impersonate this random French company in particular. Or it's just an impostor, like we saw above.\n\nWe also have the `t13d5212h1_b262b3658495_.*` group which has `BuiltWith/1.4; rb.gy/xprgqj` in the User-Agent. They have a [site describing their crawler](https://builtwith.com/biup) and similarly, I think this is associated with Google LLC only because of some shuffle of IP addresses between Google LLC ASN and GCP ASN.\n\nThe last group seems the most interesting to me. All user agents are covered by the `t13d1514h2_8daaf6152771_.*` fingerprint but you can see quite the variety of browsers and operating systems. While it is possible that all this hardware and software versions produce very similar fingerprints, my bet would be that Google is trying to see how the page behaves under different versions of Chrome for.... unknown reasons. UX? Legibility? General interest? I don't know.\n\nWhile I didn't consent to this level of scrutiny, I have to admit that it has literally no impact on the site operations, so it's all good in my book.\n\nThere are way too many bots looking at my page from Google. I really don't know why they need a bag of eyes staring at me, at this point it would be easier to just copy my page fully and then dissect it internally. They behave quite well though and it's not a problem for my site.\n\nWhile there's only one bot specifically marked for AI training (Google Other), there's nothing really preventing Google from grabbing the other bot outputs for these purposes. And as you can see, fully blocking Google is hard, even if you accept not being searchable by the most prominent search engine.\n\nAs we saw, there are several good marks you can focus on if you want to block Google. These are:\n\n`t13d181300_e8a523a41297_43ade6aba3df` JA4 Fingerprint (YMMV),`t13d1514h2_8daaf6152771_.*` JA4 Fingerprint (YMMV).\nWith that out of the way, I'll take a look at the rest of the traffic next time.\n\nThe trick is organizing IP addresses to binary trees, where the prefix is known and suffix is variable. This plays really well with the [CIDR](https://en.wikipedia.org/wiki/Subnet) notation, read more in [my ASN Check article](https://blog.miloslavhomer.cz/asn-check/#ips-and-binary-trees). Here's a quick \"illustration\":\n\n`192.168.50.102 = 11000000.10101000.00110010.01100110`\n\n192.168.50.102/24 = 11000000.10101000.00110010.????????", "url": "https://wpnews.pro/news/how-google-sees-your-site", "canonical_source": "https://blog.miloslavhomer.cz/how-google-sees-your-site/", "published_at": "2026-09-13 15:23:18+00:00", "updated_at": "2026-09-13 15:44:51.671930+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-policy"], "entities": ["Google", "HAProxy", "JA4", "marimo", "Google Other", "ASNChecker", "Uptime-Kuma"], "alternates": {"html": "https://wpnews.pro/news/how-google-sees-your-site", "markdown": "https://wpnews.pro/news/how-google-sees-your-site.md", "text": "https://wpnews.pro/news/how-google-sees-your-site.txt", "jsonld": "https://wpnews.pro/news/how-google-sees-your-site.jsonld"}}