Overwhelmed by AI Bots A self-hosted blog operator abandoned an attempt to build a simple admin guideline for separating AI bots from other bots, concluding the landscape changes too fast to maintain, and reported that roughly 8% of potential traffic matched AI bot user agents listed in Cloudflare Radar — about 20 requests per hour, a volume the operator called no concern. The operator noted that Cloudflare Radar's GitHub mirror lacks IP ranges and that bots such as Amzn-User and Kimi-User dropped out of the index despite still being listed in official documentation as of 17 November 2026, and said rate-limiting works well in the meantime while Fingerprint.js or Anubis are options for next time. The operator also cited Cloudflare's claim that even Googlebot feeds data into AI training and noted Googlebot is commonly impersonated. I've tried to analyze AI bots coming to this self-hosted blog. I wanted a simple guideline for admins to follow so we can separate AI bots from other bots and users. I give up. There are too many, changing too frequently and there’s no fixed point to hold on to. Here’s the Marimo notebook that’s WIP https://gist.github.com/ArcHound/58822d2674ffc877af9f6a703d2371c9 , probably will never be finished. By the time I fix one section, another one is outdated. As I spent way too much time on this, I'm just seeking closure with something positive. So here are my notes: Next time I'll try Fingerprint.js https://github.com/fingerprintjs/fingerprintjs or Anubis https://anubis.techaro.lol/ . Rate-limiting works well in the meantime. Let's dig in. Taking a look it seems I have approximately 8% of potential AI traffic user agent is in the list of AI bots as per Cloudflare Radar . Here’s the breakdown of that 8%: Keep in mind that this includes impostors - especially Googlebot https://developers.google.com/search/docs/crawling-indexing/googlebot is commonly impersonated and I think that’s the reason for it’s majority. In the separate Google article https://blog.miloslavhomer.cz/how-google-sees-your-site/ , I’ve counted that valid Google traffic is an order of magnitude less than trash from GCP. I’m happy to report that this volume is of no concern to my humble site. Even with this potential number, that’s about 20 requests per hour. There are three general categories of the AI bots with some differences in intent: Most companies use these categories of bots for these exact purposes. An outsider has no way of verifying any of this, we have to trust the declared intent. And it’s hard to, especially when reputable sources like Cloudflare claim that even bots like Googlebot crawling for search engine purposes feeds data into AI training. Speaking of the devil... Everybody knows TM that Cloudflare is the creme de la creme of bot detection. And they publish a lot for free too in Cloudflare Radar https://radar.cloudflare.com/ I wanted to base most of the conclusion on their data, because it’s a one-stop-shop. Then I’ve found this “mirror” https://github.com/microlinkhq/cloudflare-bot-directory/ of Cloudflare Radar on GitHub, which seemed like the perfect baseline for bot intelligence. Unfortunately, we’re missing the IP ranges and I’ve observed the bots are dropping out of the index randomly. For example, the Amzn-User bot got lost somewhere, even though official docs https://developer.amazon.com/docs/amazonbot/amazonbot.html still list it as of 17th November 2026 . Same story with Kimi-User https://www.kimi.ai/policies/kimi-crawlers . Of course, it’s a live intelligence feed so it changes in time. The format can change without prior warning and that’s fine. It’s still unpleasant though. Look I get it. It’s very hard to monitor so many bot swarms as I’ve learned , to keep all this intelligence up-to-date. Especially when the bots in question have no motive for cooperating. For starters, you have to observe an enormous amount of traffic and that’s a feat in an of itself. When getting intelligence alternatives, you usually see two issues. It's either expensive or incomplete. Here are some incomplete overviews just search for Kimi or Instapaper - I didn't find it : I'm not paying for this, so I can't recommend paid sources. I'm confident there are people happy to take your money in exchange for such intel. I saw LLMPulse https://llmpulse.ai/ai-crawler-index , but they admit as of 18th November 2026 they base this on Cloudflare Radar data. So YMMV, I conclude that Cloudflare Radar is the best. I might be very wrong about this. Maybe the real treasure were standards I’ve encountered along the way. Because I’m sure I’ll see them again someday soon. I hate parsing user agents. The standard is non-existent, the best I’ve found is this section in RFC9110 https://www.rfc-editor.org/info/rfc9110/ section-10.1.5 and that’s clearly not enough. It’s infuriating because you see two user agents that are of the same origin, but handling all of the nonsense in code is just too much work. Robots.txt RFC9309 also agrees https://www.rfc-editor.org/info/rfc9309/ name-the-user-agent-line - the rule is if a substring is present, then the user-agent matches, e.g. GPTBot matches Mozilla/5.0 AppleWebKit/537.36 KHTML, like Gecko; compatible; GPTBot/1.4; + https://openai.com/gptbot