Contents
In the thirty days ending November 2, 2026, this site logged about 13,000 requests that arrived under the names of AI companies' crawlers. Roughly 850 passed verification. About 11,800 failed it outright, and the few hundred left over neither passed nor failed cleanly.
The name is typed by the sender #
Every request carries a name the client gives itself, and the rules a site writes for crawlers are addressed to those names. That file, robots.txt, has been a formal standard since 2022, and the standard is candid about how identity works inside it: crawlers "set their own name," and the rules "are not a form of access authorization." Nothing in the arrangement checks the name. A rule written for GPTBot reaches whoever decides to call itself GPTBot.
Google-Extended is a control token, a word a site puts in robots.txt to opt out of some of Google's AI uses. No real crawler sends it as its own name. In the same thirty days, 181 requests did.
Checking happens somewhere else. Cloudflare sits in front of this site and keeps a directory of verified bots, confirmed through published address lists or a cryptographic signature. Since September 27 the site's own logs have recorded which requests passed that check, which makes it possible to set what crawlers claim beside what can be confirmed.
The plainest case is ClaudeBot, which Anthropic describes as collecting content that could contribute to model training. Anthropic's crawlers passed verification 180 times in the window. Requests that said ClaudeBot and failed came to 2,138, all but eleven of them from addresses registered to Google. Whoever sent those, it was not the crawler they named.
My earlier count was a count of names #
In August I reported that declared AI-training crawlers fetched this site 1,534 times in fourteen days, and that one of those fetches touched the rights files the site publishes. "Declared" was the accurate word, and it carried more weight than it seemed to. The verification records did not exist yet, so that figure cannot be rechecked. It counted names.
With the check in place, the two populations come apart. Over the thirty days, 144 requests passed as AI training crawlers, against about 11,800 that wore an AI company's crawler name and failed, spread across nearly every name a site might write a rule for. Read by name, the log would show AI crawlers at about fourteen times the volume that passed verification.
| Company | Passed verification | Used its crawler names and failed |
|---|---|---|
| OpenAI | 593 | 2,684 |
| Anthropic | 180 | 2,221 |
| Meta | 0 | 1,755 |
| Perplexity | 72 | 1,433 |
| Amazon | 0 | 752 |
| Apple | 0 | 709 |
| Common Crawl | 0 | 705 |
| ByteDance | 0 | 689 |
| Cohere | 0 | 655 |
| Mistral | 0 | 171 |
| Google (AI agents) | 8 | 19 |
What a site can actually confirm #
A site has two practical ways to confirm a crawler. One is the operator's published list of addresses, which Cloudflare's directory relies on. The other is a signature, through a method called Web Bot Auth.
- Web Bot Auth
- A way for a crawler to sign each request with a private key, so a site can check it against a public key the operator publishes at a known address. Cloudflare documents the method and checks the signatures.
Of the roughly 59,500 machine requests here that did not come from my own tools, 3,087 carried a valid signature, about 5 percent, and none carried an invalid one. The named signers were a few search crawlers and search-ranking tools.
For anyone keeping rights terms or measuring scraping, that changes what a log can say. A rule addressed to a name binds the crawlers honest enough to use their own, which are the least likely to need binding, and a tally kept by name mostly records who is borrowing which name. The verified figures are smaller and much less dramatic. In this window, none of the signed requests named a large AI lab's crawler.
References #
- Koster, M., Illyes, G., Zeller, H., & Sassman, L. "Robots Exclusion Protocol." RFC 9309, Internet Engineering Task Force, September 2022. rfc-editor.org/rfc/rfc9309
- Cloudflare. "Verified bots." Cloudflare bot solutions docs. Accessed November 2026. developers.cloudflare.com
- Cloudflare. "Web Bot Auth." Cloudflare bot solutions docs. Accessed November 2026. developers.cloudflare.com
- Anthropic. "Does Anthropic crawl data from the web, and how can site owners block the crawler?" Claude Help Center. Accessed November 2026. support.claude.com