# How to measure how much of your crawl traffic is AI bots, from your own access logs (in Python) > Source: > Published: 2026-11-15 14:14:19+00:00 Automated technical SEO checks that catch indexing blocks and deploy regressions — [seotto.com](https://seotto.com) Opinions about AI crawlers are everywhere. Numbers from your own site are rarer, and they're the only ones that should drive a `robots.txt` change. The questions worth answering first: Your access logs already have the answers. This post is a short Python script that classifies requests by user agent and reports exactly those numbers. User agent strings change, and operators add new ones, so treat this as a starting list and check each operator's documentation before relying on it: `Googlebot`, `bingbot`, `DuckDuckBot`, `YandexBot`, `Applebot` (the plain one, used for Apple's search features).`GPTBot`, `ClaudeBot`, `CCBot`, `Bytespider`, `meta-externalagent`, `Amazonbot`, `cohere-ai`.` OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`.` ChatGPT-User`, `Claude-User`, `Perplexity-User`. `Google-Extended` and `Applebot-Extended` won't appear in logs: they're `robots.txt` tokens, not separate crawlers. Google's AI products fetch with the normal Googlebot. Like Googlebot, all of these can be faked. A request claiming to be `GPTBot` from a residential IP is probably a scraper. The bigger operators publish their IP ranges; for numbers you'll act on, check requests against those ranges, the same way you'd verify Googlebot with reverse DNS. ``` python import gzip, glob, re, sys from collections import Counter, defaultdict CATEGORIES = { "search": ["Googlebot", "bingbot", "DuckDuckBot", "YandexBot", "Applebot/"], "ai-training": ["GPTBot", "ClaudeBot", "CCBot", "Bytespider", "meta-externalagent", "Amazonbot", "cohere-ai"], "ai-search": ["OAI-SearchBot", "PerplexityBot", "Claude-SearchBot"], "ai-user": ["ChatGPT-User", "Claude-User", "Perplexity-User"], } BOTLIKE = re.compile(r"bot|crawl|spider|slurp|fetch", re.I) # combined log format: ... "GET /path HTTP/1.1" 200 1234 "referer" "user agent" LINE = re.compile(r'"[A-Z]+ (\S+) [^"]*" (\d{3}) (\d+|-) "[^"]*" "([^"]*)"') def classify(ua): for cat, needles in CATEGORIES.items(): for n in needles: if n in ua: return cat, n if BOTLIKE.search(ua): return "other-bot", "other" return "human/other", "-" def lines(pattern): for p in sorted(glob.glob(pattern)): f = gzip.open(p, "rt", errors="replace") if p.endswith(".gz") else open(p, errors="replace") with f: yield from f reqs, bytes_, agents = Counter(), Counter(), Counter() paths = defaultdict(Counter) statuses = defaultdict(Counter) for line in lines(sys.argv[1]): m = LINE.search(line) if not m: continue path, status, size, ua = m.groups() cat, agent = classify(ua) reqs[cat] += 1 bytes_[cat] += 0 if size == "-" else int(size) agents[(cat, agent)] += 1 section = "/" + path.lstrip("/").split("/", 1)[0].split("?", 1)[0] paths[cat][section] += 1 statuses[cat][status[0] + "xx"] += 1 total_bot = sum(n for c, n in reqs.items() if c != "human/other") print(f"{'category':14s} {'requests':>10s} {'% of bots':>10s} {'MB':>10s} statuses") for cat, n in reqs.most_common(): share = "" if cat == "human/other" else f"{100 * n / max(total_bot, 1):.1f}%" print(f"{cat:14s} {n:10d} {share:>10s} {bytes_[cat] / 1e6:10.1f} {dict(statuses[cat])}") print("\nTop agents:") for (cat, agent), n in agents.most_common(15): if cat != "human/other": print(f"{n:8d} {cat:12s} {agent}") for cat in ("ai-training", "ai-search", "ai-user"): if paths[cat]: print(f"\nTop sections for {cat}:") for s, n in paths[cat].most_common(8): print(f"{n:8d} {s}") ``` Run it over a month of logs: ``` python3 ai_bots.py "/var/log/nginx/access.log*" ``` If you're behind a CDN, run it on the CDN's logs instead. Many bot requests are served from the edge cache and never reach your origin, so origin logs undercount. **Logs tell you what happened; a recurring check tells you when it changes.** [seotto.com](https://seotto.com) flags indexing and crawler-access regressions so you don't have to re-run the script by hand. On many content sites, AI-related crawlers now make up a noticeable share of all bot requests, sometimes rivalling Googlebot. On a small site with little content, they may be a rounding error. Either way, this one number tells you whether the issue is worth more of your time. Compare the MB column with the request count. A crawler with many requests but small transfers is mostly hitting cached or small pages. One with large transfers per request may be downloading images, PDFs or large HTML pages. If server cost is your concern, bandwidth is the number to watch. A lot of `4xx` for one agent usually means it's requesting URLs that don't exist: old links, guessed paths, or URLs from a stale dataset. A lot of `3xx` means it's following your redirects. Neither is harmful, but both are wasted capacity on your side. If training crawlers spend most of their requests on `/blog` and `/docs`, that's your published content being collected. If they're in `/search`, `/tag` or parameter URLs, they're stuck in the same crawl traps that waste Googlebot's time, and fixing those helps both. Requests from agents like `ChatGPT-User` are people asking an assistant about your page. The top paths here show which content people bring to AI assistants, which is useful product and content feedback in its own right. To check whether a busy "AI crawler" is genuine, take its top IP addresses from the logs and compare them with the operator's published ranges. OpenAI, for example, publishes JSON files of IP ranges for its crawlers, and other large operators document theirs. Python's `ipaddress` module makes the comparison a few lines: load the CIDR blocks, then test `ipaddress.ip_address(ip) in network` for each. Anything claiming to be a known crawler from outside its operator's ranges belongs in the "other-bot" bucket, and is often the real source of the load people blame on AI. With these numbers you can decide per category: allow, block in `robots.txt`, or rate limit at the CDN. Rerun the script a month after any change to confirm the crawlers you blocked actually stopped, and that search crawlers were unaffected. At [seotto.com](https://seotto.com) we like decisions like this to start from a log query rather than a headline. What share of your bot traffic turned out to be AI crawlers? And did any agent show up that you'd never heard of?