{"slug": "show-hn-ai-drop-project-drop-traffic-fast-from-misbehaving-crawlers-agpl", "title": "Show HN: AI-drop project – drop traffic fast from misbehaving crawlers (AGPL)", "summary": "Developer Catalin(ux) C.M.T. BOIE released AI-drop, an AGPL-licensed free-software project that uses eBPF to drop packets from misbehaving crawlers, after rocketgit.com saw 4 million requests per day instead of the expected thousands. AI-drop builds a honeypot-derived blocklist of roughly 1,573,000 IPv4 and 6,000 IPv6 addresses, keeps each entry for 7 days, and streams the list to downstream Linux servers, since IP- and user-agent-based filtering fails against crawlers using millions of IPs and fake user agents. BOIE said the approach is working on rocketgit.com and noted that no AI was used to build the software.", "body_md": "Hello!\n\nAll started with 4 million requests / day on rocketgit.com (AGPL git hosting). Instead of thousands. Enough is enough!\n\nPresentation (8 pages):\n[https://rocketgit.com/user/catalinux/ai-drop/artefacts/downl...](https://rocketgit.com/user/catalinux/ai-drop/artefacts/download/docs/pre1.pdf)\n\nSo, I built AI-drop free-software project (AGPL) to block this kind of crawlers. rocketgit.com has a very restrictive robots.txt (only around 10 pages are allowed). But the crawlers are not respecting it. So, who is not respecting it will be blocked.\n\nLimiting by IP or user-agent is not working anymore. The crawlers are using millions of IPs and it takes days to reuse an IP. And they use fake user-agents.\n\nI am using honeypots to build a list of misbehaving IPs which are streamed to downstream instances. Right now I have around 1.573.000 IPv4 and 6.000 IPv6 misbehaving addresses. I keep them in the list only for 7 days to not block somebody legit. I am watching the traffic on rocketgit.com and the solution is working nicely.\n\nIt uses an eBPF program to fast drop the packets.\n\nYou can try it easily on your Linux servers:\n\n# Check [https://rocketgit.com/user/catalinux/ai-drop/pkg](https://rocketgit.com/user/catalinux/ai-drop/pkg)\n# If your distribution is already there, follow the instructions.\n# It is the best way to keep you up-to-date.\n\n# Look inside 'ai-drop.spec' or 'debian/control' and install the dependencies.\ngit clone [https://rocketgit.com/user/catalinux/ai-drop](https://rocketgit.com/user/catalinux/ai-drop)\ncd ai-drop\n./configure\nmake install\n# edit /etc/ai-drop.conf\n# Very important:\n#       set 'interface' (non 'eth' are not supported now)\n#       set 'safe_mode = 0' if you only want to evaluate the product.\n\nThen:\n\nsystemctl enable --now ai-drop journalctl -f -u ai-drop\n\n# The list of misbehaving IPs will be streamed from the main server.\n\n# Look at the statistics, especially 'drops': ai-drop dump-stats # Output example: # now=1794482762 ipv4_total=0 ipv6_total=0 total=0 \\ # ipv4_adds=0 ipv6_adds=0 adds=0 \\ # ipv4_updates=0 ipv6_updates=0 updates=0 \\ # ipv4_drops=0 ipv6_drops=0 drops=0 \\ # ipv4_real_drops=0 ipv6_real_drops=0 real_drops=0 \\ # ipv4_expires=0 ipv6_expires=0 expires=0 \\ # ipv4_deletes=0 ipv6_deletes=0 deletes=0 \\ # last_load=0 daemon_start=0\n\nPlease note that no AI was used to build this software.\n\nThank you for your time!\n\nCatalin(ux) C.M.T. BOIE\n\nComments URL: [https://news.ycombinator.com/item?id=50479949](https://news.ycombinator.com/item?id=50479949)\n\nPoints: 2\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-ai-drop-project-drop-traffic-fast-from-misbehaving-crawlers-agpl", "canonical_source": "https://rocketgit.com/user/catalinux/ai-drop", "published_at": "2026-11-13 08:05:44+00:00", "updated_at": "2026-11-13 08:20:46.173546+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-infrastructure", "developer-tools"], "entities": ["AI-drop", "Catalin(ux) C.M.T. BOIE", "rocketgit.com", "eBPF", "Hacker News"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-ai-drop-project-drop-traffic-fast-from-misbehaving-crawlers-agpl", "markdown": "https://wpnews.pro/news/show-hn-ai-drop-project-drop-traffic-fast-from-misbehaving-crawlers-agpl.md", "text": "https://wpnews.pro/news/show-hn-ai-drop-project-drop-traffic-fast-from-misbehaving-crawlers-agpl.txt", "jsonld": "https://wpnews.pro/news/show-hn-ai-drop-project-drop-traffic-fast-from-misbehaving-crawlers-agpl.jsonld"}}