How to measure how much of your crawl traffic is AI bots, from your own access logs (in Python)

A developer published a Python script that classifies access-log requests by user agent into search, AI-training, AI-search, and AI-user crawler categories, reporting request counts, byte volume, status codes, and top sections per category. The script parses combined log format (including gzipped files), flags unlisted bot-like agents, and warns that user agents can be faked, recommending verification against operators' published IP ranges and running against CDN logs since edge caching undercounts origin traffic.

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com https://seotto.com Opinions about AI crawlers are everywhere. Numbers from your own site are rarer, and they're the only ones that should drive a robots.txt change. The questions worth answering first: Your access logs already have the answers. This post is a short Python script that classifies requests by user agent and reports exactly those numbers. User agent strings change, and operators add new ones, so treat this as a starting list and check each operator's documentation before relying on it: Googlebot , bingbot , DuckDuckBot , YandexBot , Applebot the plain one, used for Apple's search features . GPTBot , ClaudeBot , CCBot , Bytespider , meta-externalagent , Amazonbot , cohere-ai . OAI-SearchBot , PerplexityBot , Claude-SearchBot . ChatGPT-User , Claude-User , Perplexity-User . Google-Extended and Applebot-Extended won't appear in logs: they're robots.txt tokens, not separate crawlers. Google's AI products fetch with the normal Googlebot. Like Googlebot, all of these can be faked. A request claiming to be GPTBot from a residential IP is probably a scraper. The bigger operators publish their IP ranges; for numbers you'll act on, check requests against those ranges, the same way you'd verify Googlebot with reverse DNS. python import gzip, glob, re, sys from collections import Counter, defaultdict CATEGORIES = { "search": "Googlebot", "bingbot", "DuckDuckBot", "YandexBot", "Applebot/" , "ai-training": "GPTBot", "ClaudeBot", "CCBot", "Bytespider", "meta-externalagent", "Amazonbot", "cohere-ai" , "ai-search": "OAI-SearchBot", "PerplexityBot", "Claude-SearchBot" , "ai-user": "ChatGPT-User", "Claude-User", "Perplexity-User" , } BOTLIKE = re.compile r"bot|crawl|spider|slurp|fetch", re.I combined log format: ... "GET /path HTTP/1.1" 200 1234 "referer" "user agent" LINE = re.compile r'" A-Z + \S+ ^" " \d{3} \d+|- " ^" " " ^" "' def classify ua : for cat, needles in CATEGORIES.items : for n in needles: if n in ua: return cat, n if BOTLIKE.search ua : return "other-bot", "other" return "human/other", "-" def lines pattern : for p in sorted glob.glob pattern : f = gzip.open p, "rt", errors="replace" if p.endswith ".gz" else open p, errors="replace" with f: yield from f reqs, bytes , agents = Counter , Counter , Counter paths = defaultdict Counter statuses = defaultdict Counter for line in lines sys.argv 1 : m = LINE.search line if not m: continue path, status, size, ua = m.groups cat, agent = classify ua reqs cat += 1 bytes cat += 0 if size == "-" else int size agents cat, agent += 1 section = "/" + path.lstrip "/" .split "/", 1 0 .split "?", 1 0 paths cat section += 1 statuses cat status 0 + "xx" += 1 total bot = sum n for c, n in reqs.items if c = "human/other" print f"{'category':14s} {'requests': 10s} {'% of bots': 10s} {'MB': 10s} statuses" for cat, n in reqs.most common : share = "" if cat == "human/other" else f"{100 n / max total bot, 1 :.1f}%" print f"{cat:14s} {n:10d} {share: 10s} {bytes cat / 1e6:10.1f} {dict statuses cat }" print "\nTop agents:" for cat, agent , n in agents.most common 15 : if cat = "human/other": print f"{n:8d} {cat:12s} {agent}" for cat in "ai-training", "ai-search", "ai-user" : if paths cat : print f"\nTop sections for {cat}:" for s, n in paths cat .most common 8 : print f"{n:8d} {s}" Run it over a month of logs: python3 ai bots.py "/var/log/nginx/access.log " If you're behind a CDN, run it on the CDN's logs instead. Many bot requests are served from the edge cache and never reach your origin, so origin logs undercount. Logs tell you what happened; a recurring check tells you when it changes. seotto.com https://seotto.com flags indexing and crawler-access regressions so you don't have to re-run the script by hand. On many content sites, AI-related crawlers now make up a noticeable share of all bot requests, sometimes rivalling Googlebot. On a small site with little content, they may be a rounding error. Either way, this one number tells you whether the issue is worth more of your time. Compare the MB column with the request count. A crawler with many requests but small transfers is mostly hitting cached or small pages. One with large transfers per request may be downloading images, PDFs or large HTML pages. If server cost is your concern, bandwidth is the number to watch. A lot of 4xx for one agent usually means it's requesting URLs that don't exist: old links, guessed paths, or URLs from a stale dataset. A lot of 3xx means it's following your redirects. Neither is harmful, but both are wasted capacity on your side. If training crawlers spend most of their requests on /blog and /docs , that's your published content being collected. If they're in /search , /tag or parameter URLs, they're stuck in the same crawl traps that waste Googlebot's time, and fixing those helps both. Requests from agents like ChatGPT-User are people asking an assistant about your page. The top paths here show which content people bring to AI assistants, which is useful product and content feedback in its own right. To check whether a busy "AI crawler" is genuine, take its top IP addresses from the logs and compare them with the operator's published ranges. OpenAI, for example, publishes JSON files of IP ranges for its crawlers, and other large operators document theirs. Python's ipaddress module makes the comparison a few lines: load the CIDR blocks, then test ipaddress.ip address ip in network for each. Anything claiming to be a known crawler from outside its operator's ranges belongs in the "other-bot" bucket, and is often the real source of the load people blame on AI. With these numbers you can decide per category: allow, block in robots.txt , or rate limit at the CDN. Rerun the script a month after any change to confirm the crawlers you blocked actually stopped, and that search crawlers were unaffected. At seotto.com https://seotto.com we like decisions like this to start from a log query rather than a headline. What share of your bot traffic turned out to be AI crawlers? And did any agent show up that you'd never heard of?