Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com
Opinions about AI crawlers are everywhere. Numbers from your own site are rarer, and they're the only ones that should drive a robots.txt change.
The questions worth answering first:
Your access logs already have the answers. This post is a short Python script that classifies requests by user agent and reports exactly those numbers.
User agent strings change, and operators add new ones, so treat this as a starting list and check each operator's documentation before relying on it:
Googlebot, bingbot, DuckDuckBot, YandexBot, Applebot (the plain one, used for Apple's search features).GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent, Amazonbot, cohere-ai. OAI-SearchBot, PerplexityBot, Claude-SearchBot. ChatGPT-User, Claude-User, Perplexity-User.
Google-Extended and Applebot-Extended won't appear in logs: they're robots.txt tokens, not separate crawlers. Google's AI products fetch with the normal Googlebot.
Like Googlebot, all of these can be faked. A request claiming to be GPTBot from a residential IP is probably a scraper. The bigger operators publish their IP ranges; for numbers you'll act on, check requests against those ranges, the same way you'd verify Googlebot with reverse DNS.
import gzip, glob, re, sys
from collections import Counter, defaultdict
CATEGORIES = {
"search": ["Googlebot", "bingbot", "DuckDuckBot", "YandexBot", "Applebot/"],
"ai-training": ["GPTBot", "ClaudeBot", "CCBot", "Bytespider", "meta-externalagent",
"Amazonbot", "cohere-ai"],
"ai-search": ["OAI-SearchBot", "PerplexityBot", "Claude-SearchBot"],
"ai-user": ["ChatGPT-User", "Claude-User", "Perplexity-User"],
}
BOTLIKE = re.compile(r"bot|crawl|spider|slurp|fetch", re.I)
LINE = re.compile(r'"[A-Z]+ (\S+) [^"]*" (\d{3}) (\d+|-) "[^"]*" "([^"]*)"')
def classify(ua):
for cat, needles in CATEGORIES.items():
for n in needles:
if n in ua:
return cat, n
if BOTLIKE.search(ua):
return "other-bot", "other"
return "human/other", "-"
def lines(pattern):
for p in sorted(glob.glob(pattern)):
f = gzip.open(p, "rt", errors="replace") if p.endswith(".gz") else open(p, errors="replace")
with f:
yield from f
reqs, bytes_, agents = Counter(), Counter(), Counter()
paths = defaultdict(Counter)
statuses = defaultdict(Counter)
for line in lines(sys.argv[1]):
m = LINE.search(line)
if not m:
continue
path, status, size, ua = m.groups()
cat, agent = classify(ua)
reqs[cat] += 1
bytes_[cat] += 0 if size == "-" else int(size)
agents[(cat, agent)] += 1
section = "/" + path.lstrip("/").split("/", 1)[0].split("?", 1)[0]
paths[cat][section] += 1
statuses[cat][status[0] + "xx"] += 1
total_bot = sum(n for c, n in reqs.items() if c != "human/other")
print(f"{'category':14s} {'requests':>10s} {'% of bots':>10s} {'MB':>10s} statuses")
for cat, n in reqs.most_common():
share = "" if cat == "human/other" else f"{100 * n / max(total_bot, 1):.1f}%"
print(f"{cat:14s} {n:10d} {share:>10s} {bytes_[cat] / 1e6:10.1f} {dict(statuses[cat])}")
print("\nTop agents:")
for (cat, agent), n in agents.most_common(15):
if cat != "human/other":
print(f"{n:8d} {cat:12s} {agent}")
for cat in ("ai-training", "ai-search", "ai-user"):
if paths[cat]:
print(f"\nTop sections for {cat}:")
for s, n in paths[cat].most_common(8):
print(f"{n:8d} {s}")
Run it over a month of logs:
python3 ai_bots.py "/var/log/nginx/access.log*"
If you're behind a CDN, run it on the CDN's logs instead. Many bot requests are served from the edge cache and never reach your origin, so origin logs undercount.
Logs tell you what happened; a recurring check tells you when it changes. seotto.com flags indexing and crawler-access regressions so you don't have to re-run the script by hand.
On many content sites, AI-related crawlers now make up a noticeable share of all bot requests, sometimes rivalling Googlebot. On a small site with little content, they may be a rounding error. Either way, this one number tells you whether the issue is worth more of your time.
Compare the MB column with the request count. A crawler with many requests but small transfers is mostly hitting cached or small pages. One with large transfers per request may be down images, PDFs or large HTML pages. If server cost is your concern, bandwidth is the number to watch.
A lot of 4xx for one agent usually means it's requesting URLs that don't exist: old links, guessed paths, or URLs from a stale dataset. A lot of 3xx means it's following your redirects. Neither is harmful, but both are wasted capacity on your side.
If training crawlers spend most of their requests on /blog and /docs, that's your published content being collected. If they're in /search, /tag or parameter URLs, they're stuck in the same crawl traps that waste Googlebot's time, and fixing those helps both.
Requests from agents like ChatGPT-User are people asking an assistant about your page. The top paths here show which content people bring to AI assistants, which is useful product and content feedback in its own right.
To check whether a busy "AI crawler" is genuine, take its top IP addresses from the logs and compare them with the operator's published ranges. OpenAI, for example, publishes JSON files of IP ranges for its crawlers, and other large operators document theirs. Python's ipaddress module makes the comparison a few lines: load the CIDR blocks, then test ipaddress.ip_address(ip) in network for each. Anything claiming to be a known crawler from outside its operator's ranges belongs in the "other-bot" bucket, and is often the real source of the load people blame on AI.
With these numbers you can decide per category: allow, block in robots.txt, or rate limit at the CDN. Rerun the script a month after any change to confirm the crawlers you blocked actually stopped, and that search crawlers were unaffected. At seotto.com we like decisions like this to start from a log query rather than a headline.
What share of your bot traffic turned out to be AI crawlers? And did any agent show up that you'd never heard of?