Show HN: AI-drop project – drop traffic fast from misbehaving crawlers (AGPL)

Developer Catalin(ux) C.M.T. BOIE released AI-drop, an AGPL-licensed free-software project that uses eBPF to drop packets from misbehaving crawlers, after rocketgit.com saw 4 million requests per day instead of the expected thousands. AI-drop builds a honeypot-derived blocklist of roughly 1,573,000 IPv4 and 6,000 IPv6 addresses, keeps each entry for 7 days, and streams the list to downstream Linux servers, since IP- and user-agent-based filtering fails against crawlers using millions of IPs and fake user agents. BOIE said the approach is working on rocketgit.com and noted that no AI was used to build the software.

Hello All started with 4 million requests / day on rocketgit.com AGPL git hosting . Instead of thousands. Enough is enough Presentation 8 pages : https://rocketgit.com/user/catalinux/ai-drop/artefacts/downl... https://rocketgit.com/user/catalinux/ai-drop/artefacts/download/docs/pre1.pdf So, I built AI-drop free-software project AGPL to block this kind of crawlers. rocketgit.com has a very restrictive robots.txt only around 10 pages are allowed . But the crawlers are not respecting it. So, who is not respecting it will be blocked. Limiting by IP or user-agent is not working anymore. The crawlers are using millions of IPs and it takes days to reuse an IP. And they use fake user-agents. I am using honeypots to build a list of misbehaving IPs which are streamed to downstream instances. Right now I have around 1.573.000 IPv4 and 6.000 IPv6 misbehaving addresses. I keep them in the list only for 7 days to not block somebody legit. I am watching the traffic on rocketgit.com and the solution is working nicely. It uses an eBPF program to fast drop the packets. You can try it easily on your Linux servers: Check https://rocketgit.com/user/catalinux/ai-drop/pkg https://rocketgit.com/user/catalinux/ai-drop/pkg If your distribution is already there, follow the instructions. It is the best way to keep you up-to-date. Look inside 'ai-drop.spec' or 'debian/control' and install the dependencies. git clone https://rocketgit.com/user/catalinux/ai-drop https://rocketgit.com/user/catalinux/ai-drop cd ai-drop ./configure make install edit /etc/ai-drop.conf Very important: set 'interface' non 'eth' are not supported now set 'safe mode = 0' if you only want to evaluate the product. Then: systemctl enable --now ai-drop journalctl -f -u ai-drop The list of misbehaving IPs will be streamed from the main server. Look at the statistics, especially 'drops': ai-drop dump-stats Output example: now=1794482762 ipv4 total=0 ipv6 total=0 total=0 \ ipv4 adds=0 ipv6 adds=0 adds=0 \ ipv4 updates=0 ipv6 updates=0 updates=0 \ ipv4 drops=0 ipv6 drops=0 drops=0 \ ipv4 real drops=0 ipv6 real drops=0 real drops=0 \ ipv4 expires=0 ipv6 expires=0 expires=0 \ ipv4 deletes=0 ipv6 deletes=0 deletes=0 \ last load=0 daemon start=0 Please note that no AI was used to build this software. Thank you for your time Catalin ux C.M.T. BOIE Comments URL: https://news.ycombinator.com/item?id=50479949 https://news.ycombinator.com/item?id=50479949 Points: 2 Comments: 0