How Google Sees Your Site A technical analysis of HAProxy logs using JA4 TLS fingerprints, user agents, and ASN data identified Google bots that are not listed in Google's own documentation, and the author notes that robots.txt is insufficient because Google admits some of its bots will not respect those rules. The examination covers Google traffic hitting a site, including the Google Other crawler used for AI training opt-out decisions, and surfaces malformed entries in the IP fields of the logs. I keep stumbling over different Google bots in my logs. So today I'm looking at each and every Googly eye that takes a look at my site. Because when you stare at me, I'll stare back. You have the right and the means to define what Google is allowed to do on your site, whether the Google traffic you face is extensive or you'd like to opt-out of AI training check Google Other https://radar.cloudflare.com/bots/directory/google-other . Sadly, using robots.txt https://en.wikipedia.org/wiki/Robots.txt is not enough, Google admits that some of their bots will not respect https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers google-safety the rules. Previous articles were about TLS fingerprinting JA3/JA4 /ja3ja4-tls-client-fingerprinting/ and how to deploy it on HAProxy /deploying-ja4/ . In this article/ marimo https://marimo.io/ notebook full source https://gist.github.com/ArcHound/89d8158af8da45014745a3e5cf2d1688 I'll use the HAProxy logs containing IP addresses plus ASNs /asn-check/ , user agents and JA4 hashes to examine Google traffic. We'll see a couple of Google bots that are not mentioned in their docs. Let's get into it. This article doubles as a marimo https://marimo.io/ notebook of course, it's not modifiable and executable, that's a security nightmare to setup . Full source here https://gist.github.com/ArcHound/89d8158af8da45014745a3e5cf2d1688 . So to kick things off, I'll do some arbitrary global imports. There are a couple dependencies like my own asn-check source https://github.com/ArcHound/asn-check , article /asn-check/ and requests https://pypi.org/project/requests/ . python import csv from ipaddress import ip address, IPv4Network, IPv6Network, IPv4Address, IPv6Address from asn check import ASNChecker from asn check.ip binary tree import IPTree from typing import Union import requests import dataclasses from enum import Enum import re from collections import defaultdict We need logs. First, I set up HAProxy check out the previous part /deploying-ja4/ . Then I parsed the logs into something more sane, e.g. like this https://gist.github.com/ArcHound/cdecf272b3d07e402b168b0ffdf6c331 . That "format" optimizes lookups, so I've forced it to CSV for easier processing. Here's the CSV file. /documents/1/stats.csv Here's a sample of what we're working with: with open 'haproxy logs/stats.csv' as f: cr = csv.DictReader f data = row for row in cr data 0 { "ja4": "t13d591000 a33745022dd6 1f22a2ca17c4" "user agent": "Uptime-Kuma/1.23.15" "ip": "161.97.71.61" "count": "157907" } Generally, the signals to look for are IP addresses, User-Agents and the overall traffic. My new signal here is JA4, which I hope will give us some clarity on classifying these bots. Another useful signal is ASN Autonomous System Number