# How to confirm an AI crawler is causing your server load, and which bot to rate-limit first > Source: > Published: 2026-10-24 05:31:42+00:00 When the load average climbs and someone in the channel says *it is the AI crawlers*, that is a hypothesis, not a diagnosis. Before you add a single rate limit, three things have to be true: Short answer for step 3: **the bot with the highest request-time seconds per minute inside the spike window, among the bots you have decided to keep.** Not the bot with the most requests. Everything below is how to get those numbers out of a log you already have. ``` nproc uptime vmstat 1 5 ``` Read `r` (run-queue length) and `wa` (I/O wait). If `r` is consistently above the number of cores from `nproc`, requests are queueing. If `wa` is high while `us` and `sy` are low, you are waiting on disk or the database and a web-tier rate limit will not fix it — but write the number down anyway, because you will want it again in step 6. A load average of 40 on a 4-core box and a load average of 40 on a 64-core box are different incidents. Always report the ratio. `$request_time` into the access log The combined log format tells you *that* a request happened, not what it cost. Add the timing field: ``` log_format main_ext '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" $request_time'; access_log /var/log/nginx/access.log main_ext; ``` Reload, and confirm a line ends with a number: ``` tail -n 3 /var/log/nginx/access.log ``` If you cannot reload right now, skip the timing column — the script below treats it as optional and you can still rank by request count. Save this as `crawl_report.py`. Standard library only, no dependencies. ``` bash #!/usr/bin/env python3 """Per-minute, per-crawler request totals from an nginx access log. Usage: python3 crawl_report.py /var/log/nginx/access.log > crawlers.csv Expects the combined log format; the trailing $request_time field is optional. """ import csv import re import sys from collections import defaultdict MONTHS = {m: f"{i:02d}" for i, m in enumerate( "Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec".split(), start=1)} LINE = re.compile( r'^(?P\S+) \S+ \S+ \[(?P[^\]]+)\] ' r'"(?P[^"]*)" (?P\d{3}) (?P\S+) ' r'"[^"]*" "(?P[^"]*)"(?:\s+(?P[0-9.]+))?\s*$' ) # substring found in the User-Agent -> label. Extend as needed. BOTS = [ ("gptbot", "GPTBot"), ("oai-searchbot", "OAI-SearchBot"), ("chatgpt-user", "ChatGPT-User"), ("claudebot", "ClaudeBot"), ("perplexitybot", "PerplexityBot"), ("bytespider", "Bytespider"), ("ccbot", "CCBot"), ("googlebot", "Googlebot"), ("bingbot", "bingbot"), ] def label(ua): low = ua.lower() for key, token in BOTS: if token.lower() in low: return key return "other" if ua else "no-ua" def minute(ts): # 24/Oct/2026:13:22:01 +0000 -> 2026-10-24T13:22 day, mon, rest = ts.split("/", 2) year, clock = rest.split(":", 1) return f"{year}-{MONTHS[mon]}-{int(day):02d}T{clock[:5]}" def main(path): requests = defaultdict(int) seconds = defaultdict(float) errors = defaultdict(int) with open(path, errors="replace") as fh: for line in fh: m = LINE.match(line) if not m: continue key = (minute(m["ts"]), label(m["ua"])) requests[key] += 1 if m["rt"]: seconds[key] += float(m["rt"]) if m["status"].startswith("5"): errors[key] += 1 out = csv.writer(sys.stdout) out.writerow(["minute", "bot", "requests", "req_seconds", "errors_5xx"]) for key in sorted(requests): out.writerow([key[0], key[1], requests[key], round(seconds[key], 1), errors[key]]) if __name__ == "__main__": main(sys.argv[1]) ``` Run it against a rotated log that covers the incident: ``` python3 crawl_report.py /var/log/nginx/access.log.1 > crawlers.csv head -5 crawlers.csv ``` If your monitoring already samples every minute, export the load series as `minuteload1` and skip the sampler. Otherwise: ``` ( while :; do printf '%s ' "$(date -u +%Y-%m-%dT%H:%M)" cut -d' ' -f1-3 /proc/loadavg sleep 60 done ) >> /tmp/load.log ``` Leave it running for a day, then join the two files on the minute column: ``` awk 'NR==FNR { load[$1] = $2; next } FNR == 1 { next } { printf "%s %-14s req=%4d req_s=%6.1f load1=%s\n", $1, $2, $3, $4, (($1 in load) ? load[$1] : "-") }' \ /tmp/load.log crawlers.csv | sort ``` The output is per minute per bot. You want a window where `load1` is high and, in the same minutes, one bot's `req` and `req_s` jump. A bot that holds a steady 2% of traffic through the spike and through the quiet hours is not your problem, whatever its total is. `req_seconds` per minute divided by 60 gives the average number of workers that traffic kept busy. Divide the *total* across all bots and all visitors by 60 as well. If the total sits well below your worker count, nothing in the web tier explains the load — look at cron jobs, backups, or the database. This is the single cheapest way to avoid throttling the wrong thing. `req_seconds` says how much time it consumed. The paths say whether that time went to a static file or to a faceted search query. ``` # volume, query strings stripped awk -F'"' '/GPTBot/ {split($2, r, " "); print r[2]}' access.log \ | cut -d'?' -f1 | sort | uniq -c | sort -rn | head -20 # variety, query strings kept awk -F'"' '/GPTBot/ {split($2, r, " "); print r[2]}' access.log \ | sort -u | wc -l ``` A crawler hitting 400 distinct filtered URLs once each is a different problem from one hitting the same homepage 400 times: the first defeats your cache, the second does not. Also check the status codes — a crawler walking a broken pagination loop can generate thousands of 404s that cost you a database lookup each. | Signal | What it tells you | Weight it | |---|---|---| | `req_seconds` in the spike window | how much server time it consumed | high | | share of total `req_seconds` during the spike vs. during quiet hours | whether it grew, or was always there | high | | distinct URLs fetched | whether it defeats caching | medium | | 5xx and 404 counts | whether it is stressing failing paths | medium | | raw request count | how chatty it is | low — often static files | Throttle one bot first. If the load does not move, you learned something for the price of one config change, and you still have the other candidates unthrottled. Also: the User-Agent string is self-reported. Before you hard-block anything, resolve the client IP and compare it against the IP ranges the operator publishes. A spoofed UA is common enough that blocking by UA alone is unreliable. Classify by UA, keep one bucket per crawler, and let everyone else through untouched: ``` # http context map $http_user_agent $crawler_bucket { default ""; "~*GPTBot" "gptbot"; "~*ClaudeBot" "claudebot"; "~*PerplexityBot" "perplexitybot"; "~*Bytespider" "bytespider"; } limit_req_zone $crawler_bucket zone=crawler_rate:10m rate=30r/m; limit_req_status 429; limit_req_log_level notice; # server context location / { limit_req zone=crawler_rate burst=10 nodelay; # ... your existing fastcgi_pass / proxy_pass } ``` Requests whose key evaluates to an empty string are not accounted by `limit_req`, so ordinary browsers fall outside the limiter. Verify that rather than trusting it: ``` curl -s -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0' https://example.com/ curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.0' https://example.com/ ``` Fire the second one 40 times in a row; you should see `429` appear. Then `nginx -t && systemctl reload nginx` and re-run the report for the equivalent window tomorrow, with `load.log` still sampling. The 30 r/m figure is a starting point, not a recommendation. Pick a rate by looking at what the bot actually consumes and what you can serve, then move it until the load series flattens. `robots.txt` `Crawl-delay` is advisory — it is a request, not an enforcement mechanism, and it is not honoured by every crawler. Use it as a courtesy alongside a real limit, never as the only control. Outright disallow is a business decision, not a technical one: someone has to decide whether that bot's traffic has any value to you before you cut it off. For WordPress sites, [AumCrawl](https://aumcreate.com/plugins/aumcrawl?utm_source=devto&utm_medium=content&utm_campaign=devto-identify-crawler-cpu-spike) is a free plugin that shows which AI crawlers are hitting the site and lets you allow or block them per crawler, which replaces steps 2 and 6 with a UI. The measurement approach above still applies if you want to justify the decision with numbers. `nproc` and the peak load are written down together as a ratio.`wa` was checked; if the bottleneck is I/O, this article is not the fix.`req_seconds` per minute ÷ 60 is compared against worker count.