If the past few weeks are any indication, more and more open source projects are drawing lines around how much AI they will let into their workflow and codebases before it's too much. Debian just voted to allow generative AI in project contributions, while Rust adopted a tiered policy that keeps AI mostly out of the actual code. Both of these projects tackled the same underlying question.
And now, Linux, one of the biggest open source projects around, is getting hammered by scrapers, mostly AI-powered ones, sending the same repetitive requests over and over and burning through compute capacity.
Most of it has nothing to do with anyone actually writing code, btw. 🙃
This is absurd #
Konstantin Ryabitsev of the Linux Foundation has put out numbers behind a complaint he has been voicing for a while now. Fourteen of git.kernel.org's 90 CPU cores, spread across five nodes, spend every second of every day turning commits into HTML pages.
On the outside, you might think,** What's the problem with this?** Every commit in Linux's history sits out in the open, is free to clone, and predates the wave of AI tools now scraping it.
You see, all of those characteristics are precisely what makes the repository a "goldmine of learning data." It houses the mainline kernel tree, every stable release branch going back years, dozens of subsystem maintainer trees, and even pre-git history from the BitKeeper era.
What's actually absurd is the way these clankers are going about the scraping business.
Konstantin ran the numbers on it and found out that a regular clone of linux.git, going through its whole commit history locally, takes about 200 CPU-seconds of server time. Scraping the same 1.48 million commits through cgit's individual pages instead eats up 280 CPU-hours.
thisif you want to understand how those two CPU time units work.
Do that same scrape across every one of the 922 forks hosted on the server, and the total balloons to** 258,160 CPU-hours**, something like 4.6 million times more expensive than a single clone.
*All that for a worse copy of data that was already free to grab in the first place. *🤭
And that's before counting the separate URLs cgit hands out for every patch, diff, and plain-text view of each commit too, which is what pushes the number of pages a single fork exposes into the quadrillions.
Blocking them turned into an arms race. Fail2Ban and IP bans worked until bots spread across whole subnets. ASN blocks worked too, until millions of residential and mobile IPs took over instead.
Anubis came next, being erected as a proof-of-work wall bots had to solve before getting through. It worked for a while until the clankers started solving increasing levels of challenge difficulty.
Wrapping up the writeup, Konstantin notes that:
However, you should know that out of the total of the 90 cores across 5 geo distributed nodes, there are 14-16 cores that are constantly doing nothing but rendering commits for scrapers.
On average, that's 20% of our entire capacity — except the swarms descend in waves and the actual graph is a lot more spiky than a 20% flatline.
Further speculating that when the AI bubble bursts, the project will quickly see a substantial decrease in the amount of "entities" (his labeling of the scrapers), trying to feed git.kernel.org to their models.
Though he also hopes they "smarten up" and stop scraping their data in the "dumbest way possible."
If you did not know who Konstantin is, he is the director of IT infrastructure security for the Linux Foundation and one of the sysadmins who keeps kernel.org up and running.
AI is inherently greedy #
None of this compute waste is unique to git.kernel.org. It is just the clearest example we have got so far.
Feeding a model means burning through cycles on tasks that could be done in a fraction of the time, and right now nobody building these systems seems particularly bothered by that math.
The only approach these systems seem to know is the one your average billionaire runs on, say a fictional one like Carter Pewterschmidt. Already has more than enough. Still wants more. Does not particularly care how it gets there.
Stay on that path long enough, and the greed stops paying for itself. Plenty of companies are already finding out the hard way, and the numbers back it up too.
Enjoyed this update? Support independent Linux news coverage
It's FOSS has been helping people use Linux for the past 14 years. Help us stay independent from big tech. Become a Plus member, enjoy ad-free reading and get 5 eBooks.