{"slug": "how-to-confirm-an-ai-crawler-is-causing-your-server-load-and-which-bot-to-rate", "title": "How to confirm an AI crawler is causing your server load, and which bot to rate-limit first", "summary": "A developer published a method for confirming whether AI crawlers are actually responsible for server load before applying rate limits, using standard Linux tools (nproc, uptime, vmstat) to check run-queue length and I/O wait, then adding nginx's $request_time field to access logs and running a dependency-free Python script that aggregates per-minute, per-crawler request counts, request-time seconds and 5xx errors. The guidance argues the bot to rate-limit first is the one consuming the most request-time seconds per minute within the spike window among bots worth keeping, not the one with the most requests.", "body_md": "When the load average climbs and someone in the channel says *it is the AI crawlers*, that is a hypothesis, not a diagnosis. Before you add a single rate limit, three things have to be true:\n\nShort answer for step 3: **the bot with the highest request-time seconds per minute inside the spike window, among the bots you have decided to keep.** Not the bot with the most requests. Everything below is how to get those numbers out of a log you already have.\n\n```\nnproc\nuptime\nvmstat 1 5\n```\n\nRead `r` (run-queue length) and `wa` (I/O wait). If `r` is consistently above the number of cores from `nproc`, requests are queueing. If `wa` is high while `us` and `sy` are low, you are waiting on disk or the database and a web-tier rate limit will not fix it — but write the number down anyway, because you will want it again in step 6.\n\nA load average of 40 on a 4-core box and a load average of 40 on a 64-core box are different incidents. Always report the ratio.\n\n`$request_time` into the access log\nThe combined log format tells you *that* a request happened, not what it cost. Add the timing field:\n\n```\nlog_format main_ext '$remote_addr - $remote_user [$time_local] '\n                    '\"$request\" $status $body_bytes_sent '\n                    '\"$http_referer\" \"$http_user_agent\" $request_time';\n\naccess_log /var/log/nginx/access.log main_ext;\n```\n\nReload, and confirm a line ends with a number:\n\n```\ntail -n 3 /var/log/nginx/access.log\n```\n\nIf you cannot reload right now, skip the timing column — the script below treats it as optional and you can still rank by request count.\n\nSave this as `crawl_report.py`. Standard library only, no dependencies.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Per-minute, per-crawler request totals from an nginx access log.\n\nUsage:\n    python3 crawl_report.py /var/log/nginx/access.log > crawlers.csv\n\nExpects the combined log format; the trailing $request_time field is optional.\n\"\"\"\nimport csv\nimport re\nimport sys\nfrom collections import defaultdict\n\nMONTHS = {m: f\"{i:02d}\" for i, m in enumerate(\n    \"Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec\".split(), start=1)}\n\nLINE = re.compile(\n    r'^(?P<ip>\\S+) \\S+ \\S+ \\[(?P<ts>[^\\]]+)\\] '\n    r'\"(?P<req>[^\"]*)\" (?P<status>\\d{3}) (?P<size>\\S+) '\n    r'\"[^\"]*\" \"(?P<ua>[^\"]*)\"(?:\\s+(?P<rt>[0-9.]+))?\\s*$'\n)\n\n# substring found in the User-Agent -> label. Extend as needed.\nBOTS = [\n    (\"gptbot\", \"GPTBot\"),\n    (\"oai-searchbot\", \"OAI-SearchBot\"),\n    (\"chatgpt-user\", \"ChatGPT-User\"),\n    (\"claudebot\", \"ClaudeBot\"),\n    (\"perplexitybot\", \"PerplexityBot\"),\n    (\"bytespider\", \"Bytespider\"),\n    (\"ccbot\", \"CCBot\"),\n    (\"googlebot\", \"Googlebot\"),\n    (\"bingbot\", \"bingbot\"),\n]\n\ndef label(ua):\n    low = ua.lower()\n    for key, token in BOTS:\n        if token.lower() in low:\n            return key\n    return \"other\" if ua else \"no-ua\"\n\ndef minute(ts):\n    # 24/Oct/2026:13:22:01 +0000 -> 2026-10-24T13:22\n    day, mon, rest = ts.split(\"/\", 2)\n    year, clock = rest.split(\":\", 1)\n    return f\"{year}-{MONTHS[mon]}-{int(day):02d}T{clock[:5]}\"\n\ndef main(path):\n    requests = defaultdict(int)\n    seconds = defaultdict(float)\n    errors = defaultdict(int)\n\n    with open(path, errors=\"replace\") as fh:\n        for line in fh:\n            m = LINE.match(line)\n            if not m:\n                continue\n            key = (minute(m[\"ts\"]), label(m[\"ua\"]))\n            requests[key] += 1\n            if m[\"rt\"]:\n                seconds[key] += float(m[\"rt\"])\n            if m[\"status\"].startswith(\"5\"):\n                errors[key] += 1\n\n    out = csv.writer(sys.stdout)\n    out.writerow([\"minute\", \"bot\", \"requests\", \"req_seconds\", \"errors_5xx\"])\n    for key in sorted(requests):\n        out.writerow([key[0], key[1], requests[key],\n                      round(seconds[key], 1), errors[key]])\n\nif __name__ == \"__main__\":\n    main(sys.argv[1])\n```\n\nRun it against a rotated log that covers the incident:\n\n```\npython3 crawl_report.py /var/log/nginx/access.log.1 > crawlers.csv\nhead -5 crawlers.csv\n```\n\nIf your monitoring already samples every minute, export the load series as `minute<TAB>load1` and skip the sampler. Otherwise:\n\n```\n( while :; do\n    printf '%s ' \"$(date -u +%Y-%m-%dT%H:%M)\"\n    cut -d' ' -f1-3 /proc/loadavg\n    sleep 60\n  done ) >> /tmp/load.log\n```\n\nLeave it running for a day, then join the two files on the minute column:\n\n```\nawk 'NR==FNR { load[$1] = $2; next }\n     FNR == 1 { next }\n     { printf \"%s %-14s req=%4d req_s=%6.1f load1=%s\\n\",\n            $1, $2, $3, $4, (($1 in load) ? load[$1] : \"-\") }' \\\n    /tmp/load.log crawlers.csv | sort\n```\n\nThe output is per minute per bot. You want a window where `load1` is high and, in the same minutes, one bot's `req` and `req_s` jump. A bot that holds a steady 2% of traffic through the spike and through the quiet hours is not your problem, whatever its total is.\n\n`req_seconds` per minute divided by 60 gives the average number of workers that traffic kept busy. Divide the *total* across all bots and all visitors by 60 as well. If the total sits well below your worker count, nothing in the web tier explains the load — look at cron jobs, backups, or the database. This is the single cheapest way to avoid throttling the wrong thing.\n\n`req_seconds` says how much time it consumed. The paths say whether that time went to a static file or to a faceted search query.\n\n```\n# volume, query strings stripped\nawk -F'\"' '/GPTBot/ {split($2, r, \" \"); print r[2]}' access.log \\\n  | cut -d'?' -f1 | sort | uniq -c | sort -rn | head -20\n\n# variety, query strings kept\nawk -F'\"' '/GPTBot/ {split($2, r, \" \"); print r[2]}' access.log \\\n  | sort -u | wc -l\n```\n\nA crawler hitting 400 distinct filtered URLs once each is a different problem from one hitting the same homepage 400 times: the first defeats your cache, the second does not. Also check the status codes — a crawler walking a broken pagination loop can generate thousands of 404s that cost you a database lookup each.\n\n| Signal | What it tells you | Weight it | \n|---|---|---|\n| `req_seconds` in the spike window | how much server time it consumed | high | \n| share of total `req_seconds` during the spike vs. during quiet hours | whether it grew, or was always there | high | \n| distinct URLs fetched | whether it defeats caching | medium | \n| 5xx and 404 counts | whether it is stressing failing paths | medium | \n| raw request count | how chatty it is | low — often static files | \n\nThrottle one bot first. If the load does not move, you learned something for the price of one config change, and you still have the other candidates unthrottled.\n\nAlso: the User-Agent string is self-reported. Before you hard-block anything, resolve the client IP and compare it against the IP ranges the operator publishes. A spoofed UA is common enough that blocking by UA alone is unreliable.\n\nClassify by UA, keep one bucket per crawler, and let everyone else through untouched:\n\n```\n# http context\nmap $http_user_agent $crawler_bucket {\n    default               \"\";\n    \"~*GPTBot\"            \"gptbot\";\n    \"~*ClaudeBot\"         \"claudebot\";\n    \"~*PerplexityBot\"     \"perplexitybot\";\n    \"~*Bytespider\"        \"bytespider\";\n}\n\nlimit_req_zone $crawler_bucket zone=crawler_rate:10m rate=30r/m;\nlimit_req_status 429;\nlimit_req_log_level notice;\n\n# server context\nlocation / {\n    limit_req zone=crawler_rate burst=10 nodelay;\n    # ... your existing fastcgi_pass / proxy_pass\n}\n```\n\nRequests whose key evaluates to an empty string are not accounted by `limit_req`, so ordinary browsers fall outside the limiter. Verify that rather than trusting it:\n\n```\ncurl -s -o /dev/null -w '%{http_code}\\n' -A 'Mozilla/5.0' https://example.com/\ncurl -s -o /dev/null -w '%{http_code}\\n' -A 'GPTBot/1.0' https://example.com/\n```\n\nFire the second one 40 times in a row; you should see `429` appear. Then `nginx -t && systemctl reload nginx` and re-run the report for the equivalent window tomorrow, with `load.log` still sampling.\n\nThe 30 r/m figure is a starting point, not a recommendation. Pick a rate by looking at what the bot actually consumes and what you can serve, then move it until the load series flattens.\n\n`robots.txt` `Crawl-delay` is advisory — it is a request, not an enforcement mechanism, and it is not honoured by every crawler. Use it as a courtesy alongside a real limit, never as the only control. Outright disallow is a business decision, not a technical one: someone has to decide whether that bot's traffic has any value to you before you cut it off.\n\nFor WordPress sites, [AumCrawl](https://aumcreate.com/plugins/aumcrawl?utm_source=devto&utm_medium=content&utm_campaign=devto-identify-crawler-cpu-spike) is a free plugin that shows which AI crawlers are hitting the site and lets you allow or block them per crawler, which replaces steps 2 and 6 with a UI. The measurement approach above still applies if you want to justify the decision with numbers.\n\n`nproc` and the peak load are written down together as a ratio.`wa` was checked; if the bottleneck is I/O, this article is not the fix.`req_seconds` per minute ÷ 60 is compared against worker count.", "url": "https://wpnews.pro/news/how-to-confirm-an-ai-crawler-is-causing-your-server-load-and-which-bot-to-rate", "canonical_source": "https://dev.to/aumcreate/how-to-confirm-an-ai-crawler-is-causing-your-server-load-and-which-bot-to-rate-limit-first-dfh", "published_at": "2026-10-24 05:31:42+00:00", "updated_at": "2026-10-24 05:41:40.084209+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-infrastructure", "developer-tools"], "entities": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "PerplexityBot", "Bytespider", "CCBot", "nginx"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-confirm-an-ai-crawler-is-causing-your-server-load-and-which-bot-to-rate", "markdown": "https://wpnews.pro/news/how-to-confirm-an-ai-crawler-is-causing-your-server-load-and-which-bot-to-rate.md", "text": "https://wpnews.pro/news/how-to-confirm-an-ai-crawler-is-causing-your-server-load-and-which-bot-to-rate.txt", "jsonld": "https://wpnews.pro/news/how-to-confirm-an-ai-crawler-is-causing-your-server-load-and-which-bot-to-rate.jsonld"}}