{"slug": "ryabitsev-creepy-crawlies", "title": "Ryabitsev: Creepy crawlies", "summary": "Konstantin Ryabitsev reports that AI crawlers now dominate traffic to the Linux kernel repositories at git.kernel.org, with only about 2% of the roughly 6 million daily requests being legitimate. Of the daily requests, 66% are blocked by the Anubis challenge, but 33% solve it and reach the main site, indicating scrapers are expending significant computational resources to access the data.", "body_md": "Konstantin Ryabitsev has written a [blog post with\nhard numbers](https://people.kernel.org/monsieuricon/creepy-crawlies) about the impact of AI crawlers on the Linux kernel\nrepositories at [git.kernel.org](https://git.kernel.org/):\n\nToday, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.\n\nIt's impossible to tell with certainty which of these are bots and which are real humans — but chances are, if it's asking for an old commit in a random old fork, it's probably not a real developer trying to do their work.\n\nWith a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.", "url": "https://wpnews.pro/news/ryabitsev-creepy-crawlies", "canonical_source": "https://lwn.net/Articles/1091203/", "published_at": "2026-08-29 09:32:34+00:00", "updated_at": "2026-08-29 09:49:15.491570+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure"], "entities": ["Konstantin Ryabitsev", "git.kernel.org", "Anubis"], "alternates": {"html": "https://wpnews.pro/news/ryabitsev-creepy-crawlies", "markdown": "https://wpnews.pro/news/ryabitsev-creepy-crawlies.md", "text": "https://wpnews.pro/news/ryabitsev-creepy-crawlies.txt", "jsonld": "https://wpnews.pro/news/ryabitsev-creepy-crawlies.jsonld"}}