Hello!
All started with 4 million requests / day on rocketgit.com (AGPL git hosting). Instead of thousands. Enough is enough!
Presentation (8 pages):
[https://rocketgit.com/user/catalinux/ai-drop/artefacts/downl...](https://rocketgit.com/user/catalinux/ai-drop/artefacts/download/docs/pre1.pdf)
So, I built AI-drop free-software project (AGPL) to block this kind of crawlers. rocketgit.com has a very restrictive robots.txt (only around 10 pages are allowed). But the crawlers are not respecting it. So, who is not respecting it will be blocked.
Limiting by IP or user-agent is not working anymore. The crawlers are using millions of IPs and it takes days to reuse an IP. And they use fake user-agents.
I am using honeypots to build a list of misbehaving IPs which are streamed to downstream instances. Right now I have around 1.573.000 IPv4 and 6.000 IPv6 misbehaving addresses. I keep them in the list only for 7 days to not block somebody legit. I am watching the traffic on rocketgit.com and the solution is working nicely.
It uses an eBPF program to fast drop the packets.
You can try it easily on your Linux servers:
git clone https://rocketgit.com/user/catalinux/ai-drop cd ai-drop ./configure make install
Then:
systemctl enable --now ai-drop journalctl -f -u ai-drop Please note that no AI was used to build this software.
Thank you for your time!
Catalin(ux) C.M.T. BOIE
Comments URL: [https://news.ycombinator.com/item?id=50479949](https://news.ycombinator.com/item?id=50479949)
Points: 2