How to confirm an AI crawler is causing your server load, and which bot to rate-limit first

A developer published a method for confirming whether AI crawlers are actually responsible for server load before applying rate limits, using standard Linux tools (nproc, uptime, vmstat) to check run-queue length and I/O wait, then adding nginx's $request_time field to access logs and running a dependency-free Python script that aggregates per-minute, per-crawler request counts, request-time seconds and 5xx errors. The guidance argues the bot to rate-limit first is the one consuming the most request-time seconds per minute within the spike window among bots worth keeping, not the one with the most requests.

When the load average climbs and someone in the channel says it is the AI crawlers , that is a hypothesis, not a diagnosis. Before you add a single rate limit, three things have to be true: Short answer for step 3: the bot with the highest request-time seconds per minute inside the spike window, among the bots you have decided to keep. Not the bot with the most requests. Everything below is how to get those numbers out of a log you already have. nproc uptime vmstat 1 5 Read r run-queue length and wa I/O wait . If r is consistently above the number of cores from nproc , requests are queueing. If wa is high while us and sy are low, you are waiting on disk or the database and a web-tier rate limit will not fix it — but write the number down anyway, because you will want it again in step 6. A load average of 40 on a 4-core box and a load average of 40 on a 64-core box are different incidents. Always report the ratio. $request time into the access log The combined log format tells you that a request happened, not what it cost. Add the timing field: log format main ext '$remote addr - $remote user $time local ' '"$request" $status $body bytes sent ' '"$http referer" "$http user agent" $request time'; access log /var/log/nginx/access.log main ext; Reload, and confirm a line ends with a number: tail -n 3 /var/log/nginx/access.log If you cannot reload right now, skip the timing column — the script below treats it as optional and you can still rank by request count. Save this as crawl report.py . Standard library only, no dependencies. bash /usr/bin/env python3 """Per-minute, per-crawler request totals from an nginx access log. Usage: python3 crawl report.py /var/log/nginx/access.log crawlers.csv Expects the combined log format; the trailing $request time field is optional. """ import csv import re import sys from collections import defaultdict MONTHS = {m: f"{i:02d}" for i, m in enumerate "Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec".split , start=1 } LINE = re.compile r'^ ?P<ip \S+ \S+ \S+ \ ?P<ts ^\ + \ ' r'" ?P<req ^" " ?P<status \d{3} ?P<size \S+ ' r'" ^" " " ?P<ua ^" " ?:\s+ ?P<rt 0-9. + ?\s $' substring found in the User-Agent - label. Extend as needed. BOTS = "gptbot", "GPTBot" , "oai-searchbot", "OAI-SearchBot" , "chatgpt-user", "ChatGPT-User" , "claudebot", "ClaudeBot" , "perplexitybot", "PerplexityBot" , "bytespider", "Bytespider" , "ccbot", "CCBot" , "googlebot", "Googlebot" , "bingbot", "bingbot" , def label ua : low = ua.lower for key, token in BOTS: if token.lower in low: return key return "other" if ua else "no-ua" def minute ts : 24/Oct/2026:13:22:01 +0000 - 2026-10-24T13:22 day, mon, rest = ts.split "/", 2 year, clock = rest.split ":", 1 return f"{year}-{MONTHS mon }-{int day :02d}T{clock :5 }" def main path : requests = defaultdict int seconds = defaultdict float errors = defaultdict int with open path, errors="replace" as fh: for line in fh: m = LINE.match line if not m: continue key = minute m "ts" , label m "ua" requests key += 1 if m "rt" : seconds key += float m "rt" if m "status" .startswith "5" : errors key += 1 out = csv.writer sys.stdout out.writerow "minute", "bot", "requests", "req seconds", "errors 5xx" for key in sorted requests : out.writerow key 0 , key 1 , requests key , round seconds key , 1 , errors key if name == " main ": main sys.argv 1 Run it against a rotated log that covers the incident: python3 crawl report.py /var/log/nginx/access.log.1 crawlers.csv head -5 crawlers.csv If your monitoring already samples every minute, export the load series as minute<TAB load1 and skip the sampler. Otherwise: while :; do printf '%s ' "$ date -u +%Y-%m-%dT%H:%M " cut -d' ' -f1-3 /proc/loadavg sleep 60 done /tmp/load.log Leave it running for a day, then join the two files on the minute column: awk 'NR==FNR { load $1 = $2; next } FNR == 1 { next } { printf "%s %-14s req=%4d req s=%6.1f load1=%s\n", $1, $2, $3, $4, $1 in load ? load $1 : "-" }' \ /tmp/load.log crawlers.csv | sort The output is per minute per bot. You want a window where load1 is high and, in the same minutes, one bot's req and req s jump. A bot that holds a steady 2% of traffic through the spike and through the quiet hours is not your problem, whatever its total is. req seconds per minute divided by 60 gives the average number of workers that traffic kept busy. Divide the total across all bots and all visitors by 60 as well. If the total sits well below your worker count, nothing in the web tier explains the load — look at cron jobs, backups, or the database. This is the single cheapest way to avoid throttling the wrong thing. req seconds says how much time it consumed. The paths say whether that time went to a static file or to a faceted search query. volume, query strings stripped awk -F'"' '/GPTBot/ {split $2, r, " " ; print r 2 }' access.log \ | cut -d'?' -f1 | sort | uniq -c | sort -rn | head -20 variety, query strings kept awk -F'"' '/GPTBot/ {split $2, r, " " ; print r 2 }' access.log \ | sort -u | wc -l A crawler hitting 400 distinct filtered URLs once each is a different problem from one hitting the same homepage 400 times: the first defeats your cache, the second does not. Also check the status codes — a crawler walking a broken pagination loop can generate thousands of 404s that cost you a database lookup each. | Signal | What it tells you | Weight it | |---|---|---| | req seconds in the spike window | how much server time it consumed | high | | share of total req seconds during the spike vs. during quiet hours | whether it grew, or was always there | high | | distinct URLs fetched | whether it defeats caching | medium | | 5xx and 404 counts | whether it is stressing failing paths | medium | | raw request count | how chatty it is | low — often static files | Throttle one bot first. If the load does not move, you learned something for the price of one config change, and you still have the other candidates unthrottled. Also: the User-Agent string is self-reported. Before you hard-block anything, resolve the client IP and compare it against the IP ranges the operator publishes. A spoofed UA is common enough that blocking by UA alone is unreliable. Classify by UA, keep one bucket per crawler, and let everyone else through untouched: http context map $http user agent $crawler bucket { default ""; "~ GPTBot" "gptbot"; "~ ClaudeBot" "claudebot"; "~ PerplexityBot" "perplexitybot"; "~ Bytespider" "bytespider"; } limit req zone $crawler bucket zone=crawler rate:10m rate=30r/m; limit req status 429; limit req log level notice; server context location / { limit req zone=crawler rate burst=10 nodelay; ... your existing fastcgi pass / proxy pass } Requests whose key evaluates to an empty string are not accounted by limit req , so ordinary browsers fall outside the limiter. Verify that rather than trusting it: curl -s -o /dev/null -w '%{http code}\n' -A 'Mozilla/5.0' https://example.com/ curl -s -o /dev/null -w '%{http code}\n' -A 'GPTBot/1.0' https://example.com/ Fire the second one 40 times in a row; you should see 429 appear. Then nginx -t && systemctl reload nginx and re-run the report for the equivalent window tomorrow, with load.log still sampling. The 30 r/m figure is a starting point, not a recommendation. Pick a rate by looking at what the bot actually consumes and what you can serve, then move it until the load series flattens. robots.txt Crawl-delay is advisory — it is a request, not an enforcement mechanism, and it is not honoured by every crawler. Use it as a courtesy alongside a real limit, never as the only control. Outright disallow is a business decision, not a technical one: someone has to decide whether that bot's traffic has any value to you before you cut it off. For WordPress sites, AumCrawl https://aumcreate.com/plugins/aumcrawl?utm source=devto&utm medium=content&utm campaign=devto-identify-crawler-cpu-spike is a free plugin that shows which AI crawlers are hitting the site and lets you allow or block them per crawler, which replaces steps 2 and 6 with a UI. The measurement approach above still applies if you want to justify the decision with numbers. nproc and the peak load are written down together as a ratio. wa was checked; if the bottleneck is I/O, this article is not the fix. req seconds per minute ÷ 60 is compared against worker count.