{"slug": "how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-in", "title": "How to measure how much of your crawl traffic is AI bots, from your own access logs (in Python)", "summary": "A developer published a Python script that classifies access-log requests by user agent into search, AI-training, AI-search, and AI-user crawler categories, reporting request counts, byte volume, status codes, and top sections per category. The script parses combined log format (including gzipped files), flags unlisted bot-like agents, and warns that user agents can be faked, recommending verification against operators' published IP ranges and running against CDN logs since edge caching undercounts origin traffic.", "body_md": "Automated technical SEO checks that catch indexing blocks and deploy regressions — [seotto.com](https://seotto.com)\n\nOpinions about AI crawlers are everywhere. Numbers from your own site are rarer, and they're the only ones that should drive a `robots.txt` change.\n\nThe questions worth answering first:\n\nYour access logs already have the answers. This post is a short Python script that classifies requests by user agent and reports exactly those numbers.\n\nUser agent strings change, and operators add new ones, so treat this as a starting list and check each operator's documentation before relying on it:\n\n`Googlebot`, `bingbot`, `DuckDuckBot`, `YandexBot`, `Applebot` (the plain one, used for Apple's search features).`GPTBot`, `ClaudeBot`, `CCBot`, `Bytespider`, `meta-externalagent`, `Amazonbot`, `cohere-ai`.` OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`.` ChatGPT-User`, `Claude-User`, `Perplexity-User`.\n`Google-Extended` and `Applebot-Extended` won't appear in logs: they're `robots.txt` tokens, not separate crawlers. Google's AI products fetch with the normal Googlebot.\n\nLike Googlebot, all of these can be faked. A request claiming to be `GPTBot` from a residential IP is probably a scraper. The bigger operators publish their IP ranges; for numbers you'll act on, check requests against those ranges, the same way you'd verify Googlebot with reverse DNS.\n\n``` python\nimport gzip, glob, re, sys\nfrom collections import Counter, defaultdict\n\nCATEGORIES = {\n    \"search\":         [\"Googlebot\", \"bingbot\", \"DuckDuckBot\", \"YandexBot\", \"Applebot/\"],\n    \"ai-training\":    [\"GPTBot\", \"ClaudeBot\", \"CCBot\", \"Bytespider\", \"meta-externalagent\",\n                       \"Amazonbot\", \"cohere-ai\"],\n    \"ai-search\":      [\"OAI-SearchBot\", \"PerplexityBot\", \"Claude-SearchBot\"],\n    \"ai-user\":        [\"ChatGPT-User\", \"Claude-User\", \"Perplexity-User\"],\n}\nBOTLIKE = re.compile(r\"bot|crawl|spider|slurp|fetch\", re.I)\n# combined log format: ... \"GET /path HTTP/1.1\" 200 1234 \"referer\" \"user agent\"\nLINE = re.compile(r'\"[A-Z]+ (\\S+) [^\"]*\" (\\d{3}) (\\d+|-) \"[^\"]*\" \"([^\"]*)\"')\n\ndef classify(ua):\n    for cat, needles in CATEGORIES.items():\n        for n in needles:\n            if n in ua:\n                return cat, n\n    if BOTLIKE.search(ua):\n        return \"other-bot\", \"other\"\n    return \"human/other\", \"-\"\n\ndef lines(pattern):\n    for p in sorted(glob.glob(pattern)):\n        f = gzip.open(p, \"rt\", errors=\"replace\") if p.endswith(\".gz\") else open(p, errors=\"replace\")\n        with f:\n            yield from f\n\nreqs, bytes_, agents = Counter(), Counter(), Counter()\npaths = defaultdict(Counter)\nstatuses = defaultdict(Counter)\nfor line in lines(sys.argv[1]):\n    m = LINE.search(line)\n    if not m:\n        continue\n    path, status, size, ua = m.groups()\n    cat, agent = classify(ua)\n    reqs[cat] += 1\n    bytes_[cat] += 0 if size == \"-\" else int(size)\n    agents[(cat, agent)] += 1\n    section = \"/\" + path.lstrip(\"/\").split(\"/\", 1)[0].split(\"?\", 1)[0]\n    paths[cat][section] += 1\n    statuses[cat][status[0] + \"xx\"] += 1\n\ntotal_bot = sum(n for c, n in reqs.items() if c != \"human/other\")\nprint(f\"{'category':14s} {'requests':>10s} {'% of bots':>10s} {'MB':>10s}  statuses\")\nfor cat, n in reqs.most_common():\n    share = \"\" if cat == \"human/other\" else f\"{100 * n / max(total_bot, 1):.1f}%\"\n    print(f\"{cat:14s} {n:10d} {share:>10s} {bytes_[cat] / 1e6:10.1f}  {dict(statuses[cat])}\")\n\nprint(\"\\nTop agents:\")\nfor (cat, agent), n in agents.most_common(15):\n    if cat != \"human/other\":\n        print(f\"{n:8d}  {cat:12s} {agent}\")\n\nfor cat in (\"ai-training\", \"ai-search\", \"ai-user\"):\n    if paths[cat]:\n        print(f\"\\nTop sections for {cat}:\")\n        for s, n in paths[cat].most_common(8):\n            print(f\"{n:8d}  {s}\")\n```\n\nRun it over a month of logs:\n\n```\npython3 ai_bots.py \"/var/log/nginx/access.log*\"\n```\n\nIf you're behind a CDN, run it on the CDN's logs instead. Many bot requests are served from the edge cache and never reach your origin, so origin logs undercount.\n\n**Logs tell you what happened; a recurring check tells you when it changes.** [seotto.com](https://seotto.com) flags indexing and crawler-access regressions so you don't have to re-run the script by hand.\n\nOn many content sites, AI-related crawlers now make up a noticeable share of all bot requests, sometimes rivalling Googlebot. On a small site with little content, they may be a rounding error. Either way, this one number tells you whether the issue is worth more of your time.\n\nCompare the MB column with the request count. A crawler with many requests but small transfers is mostly hitting cached or small pages. One with large transfers per request may be downloading images, PDFs or large HTML pages. If server cost is your concern, bandwidth is the number to watch.\n\nA lot of `4xx` for one agent usually means it's requesting URLs that don't exist: old links, guessed paths, or URLs from a stale dataset. A lot of `3xx` means it's following your redirects. Neither is harmful, but both are wasted capacity on your side.\n\nIf training crawlers spend most of their requests on `/blog` and `/docs`, that's your published content being collected. If they're in `/search`, `/tag` or parameter URLs, they're stuck in the same crawl traps that waste Googlebot's time, and fixing those helps both.\n\nRequests from agents like `ChatGPT-User` are people asking an assistant about your page. The top paths here show which content people bring to AI assistants, which is useful product and content feedback in its own right.\n\nTo check whether a busy \"AI crawler\" is genuine, take its top IP addresses from the logs and compare them with the operator's published ranges. OpenAI, for example, publishes JSON files of IP ranges for its crawlers, and other large operators document theirs. Python's `ipaddress` module makes the comparison a few lines: load the CIDR blocks, then test `ipaddress.ip_address(ip) in network` for each. Anything claiming to be a known crawler from outside its operator's ranges belongs in the \"other-bot\" bucket, and is often the real source of the load people blame on AI.\n\nWith these numbers you can decide per category: allow, block in `robots.txt`, or rate limit at the CDN. Rerun the script a month after any change to confirm the crawlers you blocked actually stopped, and that search crawlers were unaffected. At [seotto.com](https://seotto.com) we like decisions like this to start from a log query rather than a headline.\n\nWhat share of your bot traffic turned out to be AI crawlers? And did any agent show up that you'd never heard of?", "url": "https://wpnews.pro/news/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-in", "canonical_source": "https://dev.to/seotto/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-logs-in-python-1hfd", "published_at": "2026-11-15 14:14:19+00:00", "updated_at": "2026-11-15 14:19:26.109904+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-search", "generative-engine-optimization"], "entities": ["GPTBot", "ClaudeBot", "CCBot", "PerplexityBot", "Google-Extended", "Bytespider", "OAI-SearchBot", "Googlebot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-in", "markdown": "https://wpnews.pro/news/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-in.md", "text": "https://wpnews.pro/news/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-in.txt", "jsonld": "https://wpnews.pro/news/how-to-measure-how-much-of-your-crawl-traffic-is-ai-bots-from-your-own-access-in.jsonld"}}