Last weekend two of Yandex's five data centres were hit by Ukrainian drone strikes, according to Reuters and the BBC. Within a day, Cloudflare's traffic data showed bot traffic from the Yandex network down to 0.4 percent of its recent maximum. A crawler fleet that had been hammering sites for weeks essentially vanished overnight, and the before-and-after charts made it to the top of Hacker News with over 600 points.
The war story is not the point. The point is what the charts proved: a single bot network can be a large share of your traffic, and most of us never look. AI crawler volume has been exploding. HUMAN Security's 2026 benchmark report, which analyzed more than a quadrillion interactions, found AI-driven traffic nearly tripled in 2025 and AI agent traffic grew 7,851 percent. That traffic lands on origin servers, not on Cloudflare dashboards, and someone pays for the CPU.
This piece is in two parts. First, a ten-minute way to measure what is actually hitting your server, using nothing but your access logs. Second, a decision path for what to do about it, ending with Anubis, the open-source proof-of-work proxy that now sits in front of GNOME's GitLab, kernel.org, the Arch wiki, Codeberg, and FFmpeg's infrastructure.
Full disclosure before we start: the measurement commands below are standard log analysis I use on my own servers, but I have not run Anubis in production myself. Everything about it here comes from its source code, its documentation, and the public writeups listed at the end. Treat this as researched analysis, not a war story.
Find your heaviest user agents. If your site sits behind nginx or Caddy, the access log already contains the answer. This one-liner ranks the top clients by request count:
awk -F'"' '{print $6}' /var/log/nginx/access.log \
| sort | uniq -c | sort -rn | head -20
Field 6 is the User-Agent in the default combined format. On a quiet personal site, expect something uncomfortable: your own monitoring, a couple of search engines, and a wall of names like Bytespider, meta-externalagent, Amazonbot, PerplexityBot, and anthropic-ai. Many of these identify themselves honestly. Others send a browser string and hope you don't check.
Separate AI crawlers from search engines. The distinction matters because blocking Googlebot has real costs. The community-maintained ai.robots.txt project tracks the user agent strings of AI crawlers, and Anubis ships a denylist synced from it: AI2Bot, Bytespider, Claude-Web, cohere-ai, Diffbot, DuckAssistBot, FacebookBot, FirecrawlAgent, GoogleOther, meta-externalagent, omgili, PerplexityBot, Perplexity-User, TikTokSpider, YouBot, and dozens more. Count their share like this:
grep -ciE "bytespider|perplexitybot|amazonbot|meta-externalagent|anthropic-ai|claude-web|omgili|youbot|firecrawl" \
/var/log/nginx/access.log
wc -l < /var/log/nginx/access.log
Divide the first number by the second. If AI crawlers are under 5 percent of requests, you probably need nothing more than a robots.txt update. If they are 30 percent, keep reading.
Check what they are crawling, not just how much. This is the part that decides whether a crawler is a nuisance or a threat:
awk -F'"' '$6 ~ /[Bb]ot|[Ss]pider|[Cc]rawl/ {print $2}' /var/log/nginx/access.log \
| awk '{print $2}' | sort | uniq -c | sort -rn | head -15
AI crawlers love expensive pages. On git forges that means diff and blame views. On documentation sites it means every revision of every page. Xe Iaso, the developer who built Anubis, wrote it after an Amazon crawler pulled apart a self-hosted Gitea instance exactly this way, working around the robots.txt rules meant to keep automated clients off the expensive endpoints. A crawler following every link does the equivalent of a denial of service without meaning to.
Step 1: robots.txt. Free, instant, and honored by well-behaved crawlers including most of the major AI ones, which now publish opt-out tokens. Google-Extended, GPTBot, ClaudeBot, and PerplexityBot all respect it. The problem is the crawlers that do not, and the ones that spoof a browser user agent entirely. robots.txt is a request, not a control.
Step 2: Rate limiting. nginx's limit_req or Caddy's rate limit module will cap the damage per IP. This works well against single-origin floods and poorly against distributed crawlers rotating through IP ranges, which is the pattern the big AI crawlers actually use.
Step 3: A Web Application Firewall or Cloudflare. If you are already behind Cloudflare, its bot fight mode covers much of this and you can stop reading. The cases where you cannot or will not route all your traffic through someone else's edge are exactly the cases Anubis was built for: self-hosted infrastructure, commodity hosting, and operators who do not want a dashboard or a six-figure contract.
Step 4: Anubis. The nuclear option, in the project's own words from its README: it "will result in your website being blocked from smaller scrapers and may inhibit good bots like the Internet Archive," and most sites do not need it. But for a small server melting under crawler load, it is the tool that worked.
Anubis is a single Go binary that runs as a reverse proxy between your edge server and your application. It reads a YAML policy file and classifies every request into four actions: ALLOW, DENY, CHALLENGE, or WEIGH. The clever part is what a challenge is.
The proof-of-work gate. When a request looks like a browser, meaning its user agent contains the string "Mozilla", Anubis serves an interstitial page instead of your content. That page makes the visitor's browser compute SHA-256 hashes until one starts with enough leading zero hex digits. The default difficulty is 4, which means 16 leading zero bits, roughly 65,000 expected hashes. On a modern desktop that is a second or two. On a phone, up to fifteen. On a plain HTTP client with no JavaScript engine, it is impossible, and that is the real gate: a crawler that cannot run modern JavaScript never gets a cookie and never sees your content.
The boring part that does most of the work. The default policy denies the whole ai.robots.txt denylist by name and allowlists Google, Bing, DuckDuckGo, the Internet Archive, Kagi, and other search engines it does not want to anger. Because most AI crawlers identify themselves honestly, the denylist blocks the bulk of AI traffic before any hashing happens. The proof-of-work is the tax on everything that lies about its identity.
The HTTP 200 trick. Here is the detail I find genuinely funny. When Anubis denies or challenges a request, it responds with HTTP 200 OK, not 403. The comment in the shipped config explains why: aggressive scrapers really want a 200, treat anything else as a transient error, and retry harder. Hand them a 200 with a jackal mascot instead of your article, and their state machine marks the page as fetched and moves on. Your HTML pipeline receives pseudoprofound gibberish, and the crawler does not know the difference.
The costs, stated honestly. Three things break, and you should know them before deploying:
The standard placement is Caddy or nginx on the edge, Anubis in the middle, your app behind it. With Docker Compose:
services:
anubis:
image: ghcr.io/techarohq/anubis:latest
environment:
BIND: ":3000"
TARGET: "http://app:8080"
POLICY_FNAME: "/data/cfg/botPolicy.yaml"
SERVE_ROBOTS_TXT: "true"
volumes:
- ./botPolicy.yaml:/data/cfg/botPolicy.yaml:ro
app:
image: nginx
volumes:
- ./www:/usr/share/nginx/html
The policy file is where the decisions live. A sensible starting point:
bots:
- name: well-known
path_regex: ^/.well-known/.*$
action: ALLOW
- name: robots-txt
path_regex: ^/robots.txt$
action: ALLOW
- name: feeds
path_regex: ^/(rss|atom|feed)\.xml$
action: ALLOW
- name: ai-crawlers
user_agent_regex: "(?i:bytespider|perplexity|amazonbot|meta-externalagent|anthropic-ai|claude-web|omgili|youbot)"
action: DENY
- name: generic-browser
user_agent_regex: Mozilla
action: CHALLENGE
Read top to bottom: plumbing paths pass through, known AI crawlers get the fake-200 denial, and everything claiming to be a browser gets one hash puzzle per week, since the passed-challenge cookie is valid for seven days. If you already maintain a robots.txt full of AI bot blocks, Anubis ships a robots2policy converter so you do not keep the same list twice.
The newest versions add a WEIGH action, where rules add and subtract suspicion points and thresholds turn the total into progressively harder challenges. It is a lightweight version of the request-scoring architecture every commercial bot vendor uses, evolving in the open.
For a small site or self-hosted service in October 2026, in order:
The deeper story from last weekend is uncomfortable. We built a web where the majority of requests are machines talking to machines, and the person running a two-dollar VPS is now doing traffic triage that used to be a security team's job. Anubis is what one developer built in response, and its rapid adoption by GNOME, kernel.org, and Codeberg says the problem is real. Measure your own logs before you need it.
I write about developer tools, AI infrastructure, and running your own servers every week. Subscribe, it's free.
Do you know what percentage of your traffic is bots? Run the one-liner and tell me what you found in the comments, I'm collecting numbers.
Sources: Reuters and the BBC on the Yandex data centre strikes, Cloudflare traffic data via Tom's Hardware, HUMAN Security's 2026 State of AI Traffic report, the Anubis project documentation and source at github.com/TecharoHQ/anubis, and the ai.robots.txt crawler list.