When the load average climbs and someone in the channel says it is the AI crawlers, that is a hypothesis, not a diagnosis. Before you add a single rate limit, three things have to be true:
Short answer for step 3: the bot with the highest request-time seconds per minute inside the spike window, among the bots you have decided to keep. Not the bot with the most requests. Everything below is how to get those numbers out of a log you already have.
nproc
uptime
vmstat 1 5
Read r (run-queue length) and wa (I/O wait). If r is consistently above the number of cores from nproc, requests are queueing. If wa is high while us and sy are low, you are waiting on disk or the database and a web-tier rate limit will not fix it — but write the number down anyway, because you will want it again in step 6.
A load average of 40 on a 4-core box and a load average of 40 on a 64-core box are different incidents. Always report the ratio.
$request_time into the access log
The combined log format tells you that a request happened, not what it cost. Add the timing field:
log_format main_ext '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" $request_time';
access_log /var/log/nginx/access.log main_ext;
Reload, and confirm a line ends with a number:
tail -n 3 /var/log/nginx/access.log
If you cannot reload right now, skip the timing column — the script below treats it as optional and you can still rank by request count.
Save this as crawl_report.py. Standard library only, no dependencies.
#!/usr/bin/env python3
"""Per-minute, per-crawler request totals from an nginx access log.
Usage:
python3 crawl_report.py /var/log/nginx/access.log > crawlers.csv
Expects the combined log format; the trailing $request_time field is optional.
"""
import csv
import re
import sys
from collections import defaultdict
MONTHS = {m: f"{i:02d}" for i, m in enumerate(
"Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec".split(), start=1)}
LINE = re.compile(
r'^(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] '
r'"(?P<req>[^"]*)" (?P<status>\d{3}) (?P<size>\S+) '
r'"[^"]*" "(?P<ua>[^"]*)"(?:\s+(?P<rt>[0-9.]+))?\s*$'
)
BOTS = [
("gptbot", "GPTBot"),
("oai-searchbot", "OAI-SearchBot"),
("chatgpt-user", "ChatGPT-User"),
("claudebot", "ClaudeBot"),
("perplexitybot", "PerplexityBot"),
("bytespider", "Bytespider"),
("ccbot", "CCBot"),
("googlebot", "Googlebot"),
("bingbot", "bingbot"),
]
def label(ua):
low = ua.lower()
for key, token in BOTS:
if token.lower() in low:
return key
return "other" if ua else "no-ua"
def minute(ts):
day, mon, rest = ts.split("/", 2)
year, clock = rest.split(":", 1)
return f"{year}-{MONTHS[mon]}-{int(day):02d}T{clock[:5]}"
def main(path):
requests = defaultdict(int)
seconds = defaultdict(float)
errors = defaultdict(int)
with open(path, errors="replace") as fh:
for line in fh:
m = LINE.match(line)
if not m:
continue
key = (minute(m["ts"]), label(m["ua"]))
requests[key] += 1
if m["rt"]:
seconds[key] += float(m["rt"])
if m["status"].startswith("5"):
errors[key] += 1
out = csv.writer(sys.stdout)
out.writerow(["minute", "bot", "requests", "req_seconds", "errors_5xx"])
for key in sorted(requests):
out.writerow([key[0], key[1], requests[key],
round(seconds[key], 1), errors[key]])
if __name__ == "__main__":
main(sys.argv[1])
Run it against a rotated log that covers the incident:
python3 crawl_report.py /var/log/nginx/access.log.1 > crawlers.csv
head -5 crawlers.csv
If your monitoring already samples every minute, export the load series as minute<TAB>load1 and skip the sampler. Otherwise:
( while :; do
printf '%s ' "$(date -u +%Y-%m-%dT%H:%M)"
cut -d' ' -f1-3 /proc/loadavg
sleep 60
done ) >> /tmp/load.log
Leave it running for a day, then join the two files on the minute column:
awk 'NR==FNR { load[$1] = $2; next }
FNR == 1 { next }
{ printf "%s %-14s req=%4d req_s=%6.1f load1=%s\n",
$1, $2, $3, $4, (($1 in load) ? load[$1] : "-") }' \
/tmp/load.log crawlers.csv | sort
The output is per minute per bot. You want a window where load1 is high and, in the same minutes, one bot's req and req_s jump. A bot that holds a steady 2% of traffic through the spike and through the quiet hours is not your problem, whatever its total is.
req_seconds per minute divided by 60 gives the average number of workers that traffic kept busy. Divide the total across all bots and all visitors by 60 as well. If the total sits well below your worker count, nothing in the web tier explains the load — look at cron jobs, backups, or the database. This is the single cheapest way to avoid throttling the wrong thing.
req_seconds says how much time it consumed. The paths say whether that time went to a static file or to a faceted search query.
awk -F'"' '/GPTBot/ {split($2, r, " "); print r[2]}' access.log \
| cut -d'?' -f1 | sort | uniq -c | sort -rn | head -20
awk -F'"' '/GPTBot/ {split($2, r, " "); print r[2]}' access.log \
| sort -u | wc -l
A crawler hitting 400 distinct filtered URLs once each is a different problem from one hitting the same homepage 400 times: the first defeats your cache, the second does not. Also check the status codes — a crawler walking a broken pagination loop can generate thousands of 404s that cost you a database lookup each.
| Signal | What it tells you | Weight it |
|---|---|---|
req_seconds in the spike window |
how much server time it consumed | high |
share of total req_seconds during the spike vs. during quiet hours |
whether it grew, or was always there | high |
| distinct URLs fetched | whether it defeats caching | medium |
| 5xx and 404 counts | whether it is stressing failing paths | medium |
| raw request count | how chatty it is | low — often static files |
Throttle one bot first. If the load does not move, you learned something for the price of one config change, and you still have the other candidates unthrottled.
Also: the User-Agent string is self-reported. Before you hard-block anything, resolve the client IP and compare it against the IP ranges the operator publishes. A spoofed UA is common enough that blocking by UA alone is unreliable.
Classify by UA, keep one bucket per crawler, and let everyone else through untouched:
map $http_user_agent $crawler_bucket {
default "";
"~*GPTBot" "gptbot";
"~*ClaudeBot" "claudebot";
"~*PerplexityBot" "perplexitybot";
"~*Bytespider" "bytespider";
}
limit_req_zone $crawler_bucket zone=crawler_rate:10m rate=30r/m;
limit_req_status 429;
limit_req_log_level notice;
location / {
limit_req zone=crawler_rate burst=10 nodelay;
}
Requests whose key evaluates to an empty string are not accounted by limit_req, so ordinary browsers fall outside the limiter. Verify that rather than trusting it:
curl -s -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0' https://example.com/
curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.0' https://example.com/
Fire the second one 40 times in a row; you should see 429 appear. Then nginx -t && systemctl reload nginx and re-run the report for the equivalent window tomorrow, with load.log still sampling.
The 30 r/m figure is a starting point, not a recommendation. Pick a rate by looking at what the bot actually consumes and what you can serve, then move it until the load series flattens.
robots.txt Crawl-delay is advisory — it is a request, not an enforcement mechanism, and it is not honoured by every crawler. Use it as a courtesy alongside a real limit, never as the only control. Outright disallow is a business decision, not a technical one: someone has to decide whether that bot's traffic has any value to you before you cut it off.
For WordPress sites, AumCrawl is a free plugin that shows which AI crawlers are hitting the site and lets you allow or block them per crawler, which replaces steps 2 and 6 with a UI. The measurement approach above still applies if you want to justify the decision with numbers.
nproc and the peak load are written down together as a ratio.wa was checked; if the bottleneck is I/O, this article is not the fix.req_seconds per minute ÷ 60 is compared against worker count.