AI crawlers are putting more pressure on Kernel.org’s infrastructure as the project now uses about 20% of its total CPU power just to handle automated scrapers.
Konstantin Ryabitsev, who helps run Kernel.org’s infrastructure, says git.kernel.org gets about 6 million requests each day for individual commit pages. Anubis proof-of-work blocks about two-thirds, but many requests still get through.
Expectably, this has a big impact. Kernel.org runs 90 CPU cores across five locations, and about 14 to 16 of them are always busy turning Git commits into HTML for scrapers.
“We spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.”
What’s frustrating is that the data being scraped is already free to access through Git. Instead of cloning repositories and working with them locally, many crawlers ask for each commit page one at a time.
The problem is made worse by how git.kernel.org is set up. The main Linux repository has about 1.48 million commits, and the site hosts hundreds of forks. With all the different views for commits, patches, diffs, and plain text, there are a huge number of URLs for crawlers to access.
Kernel.org has tried different ways to control the traffic, like blocking certain IPs and networks. However, these steps became less effective as crawlers started using residential and mobile proxy networks.
The team later set up Anubis, which makes visitors solve a small proof-of-work challenge before they can use the site. Unfortunately, as can be expected, crawlers soon adapted and started solving even harder challenges.
In light of those events, Kernel.org is now considering a more direct approach. They may cut down the number of crawlable URLs and limit resource-heavy actions for anonymous users. Of course, the Linux kernel repositories and development data will still be open to the public.
For additional details, see Ryabitsev’s post.