cd /news/ai-crawlers/most-of-the-ai-crawlers-were-somebod… · home › topics › ai-crawlers › article
[ARTICLE · art-173895] src=juanlentino.com ↗ pub= topic=ai-crawlers verified=true sentiment=· neutral

Most of the AI crawlers were somebody else

A 30-day log of about 13,000 requests claiming to be AI company crawlers on juanlentino.com found roughly 850 passed Cloudflare's verified-bot check while about 11,800 failed it, according to the site's operator. Requests naming Anthropic's ClaudeBot failed verification 2,138 times, all but eleven from addresses registered to Google, against 180 verified ClaudeBot requests; OpenAI's names drew 593 verified and 2,684 failed requests, and Meta, Amazon, Apple, Common Crawl, ByteDance, Cohere and Mistral each had zero verified requests under their crawler names. The operator said the finding means robots.txt rules and name-based crawler tallies mostly record who is borrowing which name, since only 3,087 of roughly 59,500 machine requests carried a valid Web Bot Auth signature.

recommendation

Verify crawler identity against published IP lists or Web Bot Auth signatures before honoring robots.txt rules or counting AI traffic; in this log about 11,800 of 13,000 named AI crawler requests failed verification.

more in playbook: Securing agents and AI tools
read4 min views3 publishedNov 21, 2026
Most of the AI crawlers were somebody else
Image: Juanlentino (auto-discovered)

Contents

In the thirty days ending November 2, 2026, this site logged about 13,000 requests that arrived under the names of AI companies' crawlers. Roughly 850 passed verification. About 11,800 failed it outright, and the few hundred left over neither passed nor failed cleanly.

The name is typed by the sender #

Every request carries a name the client gives itself, and the rules a site writes for crawlers are addressed to those names. That file, robots.txt, has been a formal standard since 2022, and the standard is candid about how identity works inside it: crawlers "set their own name," and the rules "are not a form of access authorization." Nothing in the arrangement checks the name. A rule written for GPTBot reaches whoever decides to call itself GPTBot.

Google-Extended is a control token, a word a site puts in robots.txt to opt out of some of Google's AI uses. No real crawler sends it as its own name. In the same thirty days, 181 requests did.

Checking happens somewhere else. Cloudflare sits in front of this site and keeps a directory of verified bots, confirmed through published address lists or a cryptographic signature. Since September 27 the site's own logs have recorded which requests passed that check, which makes it possible to set what crawlers claim beside what can be confirmed.

The plainest case is ClaudeBot, which Anthropic describes as collecting content that could contribute to model training. Anthropic's crawlers passed verification 180 times in the window. Requests that said ClaudeBot and failed came to 2,138, all but eleven of them from addresses registered to Google. Whoever sent those, it was not the crawler they named.

My earlier count was a count of names #

In August I reported that declared AI-training crawlers fetched this site 1,534 times in fourteen days, and that one of those fetches touched the rights files the site publishes. "Declared" was the accurate word, and it carried more weight than it seemed to. The verification records did not exist yet, so that figure cannot be rechecked. It counted names.

With the check in place, the two populations come apart. Over the thirty days, 144 requests passed as AI training crawlers, against about 11,800 that wore an AI company's crawler name and failed, spread across nearly every name a site might write a rule for. Read by name, the log would show AI crawlers at about fourteen times the volume that passed verification.

Company Passed verification Used its crawler names and failed
OpenAI 593 2,684
Anthropic 180 2,221
Meta 0 1,755
Perplexity 72 1,433
Amazon 0 752
Apple 0 709
Common Crawl 0 705
ByteDance 0 689
Cohere 0 655
Mistral 0 171

| Google (AI agents) | 8 | 19 |

What a site can actually confirm #

A site has two practical ways to confirm a crawler. One is the operator's published list of addresses, which Cloudflare's directory relies on. The other is a signature, through a method called Web Bot Auth.

  • Web Bot Auth
  • A way for a crawler to sign each request with a private key, so a site can check it against a public key the operator publishes at a known address. Cloudflare documents the method and checks the signatures.

Of the roughly 59,500 machine requests here that did not come from my own tools, 3,087 carried a valid signature, about 5 percent, and none carried an invalid one. The named signers were a few search crawlers and search-ranking tools.

For anyone keeping rights terms or measuring scraping, that changes what a log can say. A rule addressed to a name binds the crawlers honest enough to use their own, which are the least likely to need binding, and a tally kept by name mostly records who is borrowing which name. The verified figures are smaller and much less dramatic. In this window, none of the signed requests named a large AI lab's crawler.

References #

  • Koster, M., Illyes, G., Zeller, H., & Sassman, L. "Robots Exclusion Protocol." RFC 9309, Internet Engineering Task Force, September 2022. rfc-editor.org/rfc/rfc9309
  • Cloudflare. "Verified bots." Cloudflare bot solutions docs. Accessed November 2026. developers.cloudflare.com
  • Cloudflare. "Web Bot Auth." Cloudflare bot solutions docs. Accessed November 2026. developers.cloudflare.com
  • Anthropic. "Does Anthropic crawl data from the web, and how can site owners block the crawler?" Claude Help Center. Accessed November 2026. support.claude.com
── more in #ai-crawlers 4 stories · sorted by recency
── more on @cloudflare 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/most-of-the-ai-crawl…] indexed:0 read:4min 2026-11-21 · —