The robots.txt user-agent lines that do not name what they look like

582 of the 2,725 sites with User-agent lines carry at least one value that is not a valid product token, so it does not name the crawler the author meant. AI2Bot is read as AI. Measured across the top 5,000 sites.

A robots.txt User-agent value is a product token, and RFC 9309 defines the characters it may contain as letters, - and _. There are no digits in that grammar and no spaces. Google's reference parser implements it the same way: ExtractUserAgent reads characters while they are in [a-zA-Z_-] and stops at the first one that is not, then compares what it got against the crawler's own name exactly, ignoring case.

So User-agent: AI2Bot is read as AI. The digit ends the token. The line still parses, it just names a crawler called AI rather than the one the author had in mind. This is a separate scan from the crawler census on the data page, so its totals differ slightly: 2,725 sites here published a robots.txt containing at least one User-agent line, and 582 of those (21.4%) carry at least one value that is not a valid token. There were 4,781 such values in total, 1,196 of them distinct.

What that means in practice splits three ways. 4,022 values on 559 sites become a token that names nothing else in the file, so the rule sits there applying to a crawler that does not exist. 568 values on 141 sites truncate into a token that IS declared elsewhere in the same file, which quietly merges two rule sets. 182 values on 120 sites reduce to nothing at all.

One honest limit on all of this. The grammar above is what the standard says and what Google's parser does. Whether the crawler named AI2Bot honours the line anyway depends on that crawler's own parser, which I cannot observe from the outside. What is certain is that anything built on the reference implementation does not read it as AI2Bot. Treat these lines as unreliable rather than as proven dead.

The twenty most common ones #

| `008` | `(nothing)` | 46 | 
| `360Spider` | `(nothing)` | 44 | 

| Crawl4AI | Crawl | 43 | | panscient.com | panscient | 43 | | peer39_crawler | peer | 42 | | iaskspider/2.0 | iaskspider | 42 | | Echobot Bot | Echobot | 42 | | Sogou web spider | Sogou | 38 | | Channel3Bot | Channel | 38 | | sitecheck.internetseer.com | sitecheck | 38 |

Counted across the Tranco top 5,000 on 2026-09-07. Each "reads as" is recomputed with the same parser the checker runs, not copied from the scan.

The same check on a list a lot of sites paste in #

The ai.robots.txt project maintains a list of AI crawlers and publishes a generated robots.txt from it. Plenty of sites use that file as it comes, which makes it worth running the grammar over. Of its 175 entries, 21 are not valid product tokens, so under the reference parser those lines do not name the crawler the list is aiming at.

4 pairs also collapse into the same token, so two entries become one group:

• AI2Bot andAI2Bot-DeepResearchEval both reduce toAI

• Brightbot andBrightbot 1.0 both reduce toBrightbot

• iaskspider andiaskspider/2.0 both reduce toiaskspider

• MistralAI-User andMistralAI-User/1.0 both reduce toMistralAI-User Three of those pairs are one name listed twice, once bare and once with a version suffix, which is harmless. AI2Bot and AI2Bot-DeepResearchEval are different crawlers that both reduce to AI, which is not.

This is not carelessness by that project. The list records the names crawlers publish for themselves, and several of those names contain digits, spaces or dots that the RFC 9309 grammar has no room for. The gap is between the names vendors choose and what the standard can express. The same caveat as above applies, so the entries are worth keeping: a crawler matching on the full string will still honour them.

What to do about it #

Do not delete these lines. A line that the reference parser reads as AI is still honoured by any crawler that matches on the full string, and removing it can only make things worse. The useful move is to stop relying on it. If there is a crawler you genuinely need to keep out, express it in a way the grammar can carry: block broadly with a catch-all Disallow and allow back the crawlers you do want by their exact token.

Version suffixes are the safe case. User-agent: CCBot/2.0 reduces to CCBot, which is the crawler the author meant, so it works. The same is true of ChatGPT-User/2.0. Writing the bare token is clearer, but nothing breaks.

You can check any site with [the checker](/), which uses this same parser, or from the command line with [aicheck](https://github.com/SamHartleyFixes/aicheck).

Related: the [5,000-site crawler census](/data), [every site that blocks an AI search crawler](/sites-blocking-ai-search), and [what blocking GPTBot actually controls](/guides/should-i-block-gptbot).