The goal here is simple. Most sysadmins are currently loosening their firewall rules or adding exceptions for AI bots to ensure their content is discoverable by the latest models. Attackers know this. By mimicking ClaudeBot, they can slip past basic filters that would normally flag a high-frequency scanner.
How to tell if your ClaudeBot traffic is fake #
Real AI crawlers generally follow a predictable pattern. They hit the homepage, follow links, and respect robots.txt
. Vulnerability scanners, however, go straight for the "expensive" or sensitive endpoints. If you see "ClaudeBot" requesting /phpmyadmin
, /.env
, or /wp-admin
at a rate of 50 requests per second, it's not an AI—it's a script.
To verify if the traffic is actually coming from Anthropic, you should perform a reverse DNS lookup. A legitimate ClaudeBot request will resolve back to a domain owned by Anthropic. If the IP resolves to a random VPS provider in a region where you have no business, it's a spoof.
A practical tutorial for filtering fake bots #
If you want to stop these scans without blocking actual AI agents, you can't rely on the User-Agent alone. You need a more robust AI workflow for your security layer. Here is a basic approach using Nginx to flag suspicious bot behavior.
- Create a map to identify the claimed bot:
map $http_user_agent $is_claude {
default 0;
"~*ClaudeBot" 1;
}
- Set up a rate limit specifically for these agents to prevent them from hammering your API:
limit_req_zone $binary_remote_addr zone=bot_limit:10m rate=1r/s;
server {
location / {
if ($is_claude) {
limit_req zone=bot_limit burst=5 nodelay;
}
try_files $uri $uri/ /index.php?$query_string;
}
}
- Use a script to cross-reference the IP with known Anthropic IP ranges. Since the IP lists change, you should automate this check via a cron job that updates your firewall rules.
For those implementing a more advanced deployment, integrating a WAF (Web Application Firewall) that supports behavioral analysis is the real-world solution. A legitimate bot doesn't try to perform SQL injection on your login page. By combining User-Agent verification with path-based anomaly detection, you can maintain the visibility you want for LLM agents while shutting out the noise of mass scans.
Anthropic is fighting the invisible watermark war 3h ago
Anthropic is building a massive data center fleet on someone 3h ago
Anthropic is finally adding invisible watermarks to its model 6h ago
Anthropic says Claude marks AI content but won't show the method 8h ago
AI agents might actually solve the GPU heat crisis 10h ago
LLMs are starting to ignore their system prompts and we need 15h ago
Next Trunchbull lets you run LLM benchmarks in a browser →