AI Scrapers Just Took Down Gentoo's Bug Tracker

Early Friday morning, Gentoo developer Michał Górny pulled the distro's Bugzilla offline because AI scrapers using thousands of different IPv4 addresses with no obvious patterns had made it unusable for humans, though the tracker returned within hours. Górny wrote, "No point in feeding the LLM scrapers that are using thousands of different IPv4 addresses, with no obvious patterns I can see," and noted he is not a sysadmin and shouldn't have to be one just to get project work done. The takedown highlights that identification-based defenses like user-agent blocklists, ASN bans, and rate limits are dead against residential-proxy crawler fleets, and even proof-of-work challenges like Anubis are being solved by major scraping operations by late 2025.

AI https://sourcefeed.dev/c/ai Article AI Scrapers Just Took Down Gentoo's Bug Tracker Residential-proxy crawler fleets have outlived every identification-based defense, and volunteer infrastructure is running out of moves. Mariana Souza https://sourcefeed.dev/u/mariana souza Early Friday morning, Gentoo https://www.gentoo.org/ developer Michał Górny pulled the distro's Bugzilla offline. Not for maintenance, not because of a breach — because it had stopped being usable for humans. "No point in feeding the LLM scrapers that are using thousands of different IPv4 addresses, with no obvious patterns I can see," he wrote, adding that he's not a sysadmin and shouldn't have to be one just to get project work done. The tracker came back within hours, but the takedown itself is the story: a maintainer of one of the oldest Linux distributions decided that the only winning move against AI crawlers was to turn the service off. If you've been following this fight since early 2025, that sentence about "thousands of different IPv4 addresses, with no obvious patterns" should stop you cold. It means the arms race has entered its endgame phase — and the defenders' entire toolkit was built for the previous one. Every identification-based defense is now dead The first wave of AI scraping, circa 2024, was rude but legible. Crawlers announced themselves in user-agent strings, came from datacenter IP ranges, and could at least theoretically be governed by robots.txt. The countermeasures matched: user-agent blocklists, ASN bans, rate limits per IP. When SourceHut https://sourcehut.org/ 's Drew DeVault wrote in March 2025 that LLM crawlers were causing weekly outage-grade incidents and that he was spending a substantial fraction of his time on mitigation, the crawlers he described were already cycling IPs — but blocking entire clouds still helped. Górny described the second wave in an April blog post: distributed traffic with "no throttling, no caching, no respect for the rules," hammering Bugzilla's search endpoint over and over. "It is like a constant DDoS attack at independent infrastructure." That's not a metaphor. A fleet spread across thousands of residential-proxy IPs, each making a handful of requests with plausible browser headers, is structurally indistinguishable from a botnet — and from legitimate users, which is the point. There is no header to match, no ASN to ban, no rate limit that triggers. Identification has failed as a strategy. That failure also bounds what Cloudflare's much-publicized default block on AI crawlers https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ can accomplish. Flipped on in July 2025 alongside a pay-per-crawl marketplace, it governs the crawlers willing to be governed — the OpenAIs and Anthropics that declare themselves and increasingly negotiate for access. The traffic that took down Gentoo's Bugzilla is the other kind: operators nobody can name, laundered through residential proxies, scraping for training corpora or agent pipelines that will never show up in a compensation marketplace. The compliant half of the ecosystem got a business model; the defiant half got a smokescreen, because now every self-identified bot is one of the good ones. Proof-of-work bought a year, not a decade The fallback, once you can't identify bots, is to tax everyone. Anubis https://anubis.techaro.lol/ , Xe Iaso's MIT-licensed proof-of-work proxy, became FOSS infrastructure's default answer in 2025 — deployed in front of GNOME's GitLab, the kernel's mailing-list archives, FFmpeg, Wine, the Arch wiki, even UNESCO. The numbers it exposed were damning: GNOME sysadmin Bart Piotrowski reported that only about 3.2% of requests passed the challenge, meaning nearly everything hitting the server was automated. But proof-of-work is an economic argument, not a wall, and the economics favor the attacker. By late 2025 major scraping operations were solving Anubis challenges — a headless Chrome fleet computes hashes just fine, and the cost lands on someone's stolen residential bandwidth anyway. Meanwhile the tax falls hardest on legitimate edge cases: text browsers, old hardware, users with JavaScript off, accessibility tooling. Górny made exactly this point in April — every countermeasure "can range from annoyance and a waste of energy to a complete accessibility blocker." Gentoo, notably, wasn't even hiding behind Anubis when this hit. But projects that were are discovering it bought them months, not a solution. There's a bitter irony in which service buckled. Bug trackers are close to the highest-value scraping target on the open web right now: structured problem → diagnosis → fix chains are precisely the data that makes coding models good. And Bugzilla is close to the worst-positioned software to serve that demand — a server-rendered Perl application where every search URL is a fresh database query. The scrapers aren't mirroring pages; they're enumerating search permutations against a CGI app designed for 2004-era traffic. Gentoo even publishes bot-access policies and structured endpoints for exactly this purpose. The crawlers don't care. Bulk-friendly access only works on adversaries who bother to look for it. What to actually do if you run public dev infrastructure Assume you're next, and triage by endpoint cost, not by bot identity: Auth-wall the expensive paths. Search, git blame , log and diff views, faceted queries. Anonymous users get static, cached, boring pages; sessions get the database. This is the single highest-leverage change, and it's where most forges are quietly heading. Pre-render the hot read paths. A bug's canonical page can be a cached static artifact regenerated on write. If a scraper wants to pull a million pages of HTML from a CDN cache, let it — that fight costs you almost nothing. Deploy Anubis anyway , but treat it as a rate limiter with a countdown clock, not a perimeter. Skip IP and country blocks. Against residential proxies you're mostly banning your own users. The unhappy conclusion is that anonymous read access to dynamic services is ending, and not because anyone chose it. Every failed defense ratchets toward login walls, and login walls push small projects toward platforms big enough to absorb the abuse — which mostly means GitHub, whose owner is also training models on the result. The scraping economy externalizes its costs onto volunteer infrastructure, then offers centralization as the cure. Gentoo — a project that banned AI-generated contributions back in April 2024, before most of the ecosystem took the question seriously — spent Friday morning offline anyway. That asymmetry is the whole story: opting out was never on the menu, and the projects with the least money are paying the largest share of the bill. Sources & further reading - Gentoo bugzilla closed due AI bot scraper overload https://social.treehouse.systems/@mgorny/117058483039362779 — social.treehouse.systems - Gentoo bugzilla closed due AI bot scraper overload https://news.ycombinator.com/item?id=49221864 — news.ycombinator.com - The pinnacle of enshittification, or Large Language Models https://blogs.gentoo.org/mgorny/2026/04/05/the-pinnacle-of-enshittification-or-large-language-models/ — blogs.gentoo.org - Content Independence Day: no AI crawl without compensation https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ — blog.cloudflare.com - Open Source Devs Say AI Crawlers Dominate Traffic, Forcing Blocks On Entire Countries https://tech.slashdot.org/story/25/03/26/016244/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries — tech.slashdot.org - The Open-Source Software Saving the Internet From AI Bot Scrapers https://news.slashdot.org/story/25/07/07/2146228/the-open-source-software-saving-the-internet-from-ai-bot-scrapers — news.slashdot.org Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.