cd /news/artificial-intelligence/ryabitsev-creepy-crawlies · home topics artificial-intelligence article
[ARTICLE · art-115013] src=lwn.net ↗ pub= topic=artificial-intelligence verified=true sentiment=· neutral

Ryabitsev: Creepy crawlies

Konstantin Ryabitsev reports that AI crawlers now dominate traffic to the Linux kernel repositories at git.kernel.org, with only about 2% of the roughly 6 million daily requests being legitimate. Of the daily requests, 66% are blocked by the Anubis challenge, but 33% solve it and reach the main site, indicating scrapers are expending significant computational resources to access the data.

read1 min views1 publishedAug 29, 2026

Konstantin Ryabitsev has written a blog post with hard numbers about the impact of AI crawlers on the Linux kernel

repositories at git.kernel.org: Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

It's impossible to tell with certainty which of these are bots and which are real humans — but chances are, if it's asking for an old commit in a random old fork, it's probably not a real developer trying to do their work.

With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @konstantin ryabitsev 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/ryabitsev-creepy-cra…] indexed:0 read:1min 2026-08-29 ·