{"slug": "an-update-on-the-scraper-situation", "title": "An Update on the scraper situation", "summary": "The problem of AI scraper bots overwhelming websites with traffic has intensified since early 2025, with attacks now coming from millions of unique IP addresses via residential proxy networks. These networks, operated by both criminal groups and companies like Bright Data, compromise devices without owners' knowledge to scrape data for training large language models, making the open web increasingly difficult to maintain.", "body_md": "# An update on the scraper situation\n\n## [LWN subscriber-only content]\n\n[Fighting the AI scraper bot scourge](/Articles/1008897/)\", published in early 2025, discussed the problem of widespread scraping of web sites in search of training data for large language models and related projects. This activity overwhelms sites with traffic. Over a year after that article is published, the problem is still growing. The hammering of sites by shadowy actors has reached new heights, and the open web is becoming increasingly difficult to maintain. Where is this traffic coming from, and what can be done about it?\n\n#### Residential proxies\n\nAs was described last year, scraper attacks come from a huge number of sources across the net. It is not unusual to see coordinated requests from millions of unique IP addresses over the course of a few hours, each of which hits the site at most two or three times. Attacker-controlled data, such as the user-agent field, is entirely fictional; each hit is meant to look like just another human with a web browser. There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.\n\nThis traffic comes predominantly from residential and mobile networks, directed by central command-and-control nodes. Software is installed on ordinary systems that takes orders from a control node, fetches web pages on demand, and forwards the resulting data back to the controller. Much of the time, this activity occurs without the knowledge or consent of the owner of the device in question. The term \"residential proxies\" is used to describe systems that are used in this way.\n\nKeep up with Linux and free softwarewith[a free trial subscription to LWN], no credit card required.\n\nThere are a few different (on the surface, at least) types of operator\nrunning residential-proxy networks to attack web sites. One type is purely\ncriminal, running scrapers on systems that have been compromised with some\nsort of malware. At the beginning of the year, Google [acted\nto take down a bot network called IPIDEA](https://cloud.google.com/blog/topics/threat-intelligence/disrupting-largest-residential-proxy-network) and provided a lot of\ninformation about how these operations work. The shutdown of IPIDEA\ncorrelated with a significant reduction in scraper traffic here at LWN;\nthings were relatively peaceful for a few months. That period of peace has\nsince come to an end, though.\n\nMore recently, media-streaming devices have been [identified](https://krebsonsecurity.com/2026/06/popa-botnet-linked-to-publicly-traded-israeli-firm/)\nas a major carrier of malicious scraping software. Sometimes the devices\nare compromised at the source; other times, they are just poorly secured\nand easily compromised after the fact.\n\nThe second sort of operator works more overtly, pretending to a degree of legitimacy and offering \"ethically sourced\" IP addresses. A company called Bright Data is one of the most prominent of these; it happily advertises its prowess at getting around web-site access controls and traffic limits. Bright Data offers a \"free\" VPN service; all that is needed is for the user to give Bright Data the ability to route traffic through the user's device — to become a part of the company's residential-proxy network, in other words. Every phone or other device that makes use of this VPN becomes yet another endpoint that will be used to attack web sites.\n\nThere are many other examples of this type of operator out there; often they offer a library that app developers can link into their offerings and be paid for hijacking their users' network connections. One of them even sent us a query about running an ad for its SDK on LWN; that was, it suffices to say, a short conversation. In general, these companies range from those that aspire toward some appearance of legitimacy, advertising \"GDPR compliance\" for example, to others that are just overtly sleazy.\n\nWhile these residential-proxy networks are used for web-site scraping, it is worth emphasizing that these operators have the ability to run code that accesses resources on whatever networks millions of devices happen to be connected to. To assume that this type of access would only be used for scraping would be naive at best.\n\nThen, of course, there are the high-profile companies developing models as\ntheir core business. These companies do their own scraping; the traffic\nthat can be easily attributed to them is clearly identified in the\nuser-agent field and, as a general rule, observes measures like\n`robots.txt`. They, too, will scrape an entire site, repeatedly,\nseemingly on the theory that articles written in 2003 might somehow have\nchanged in the last day, but they do not generate overwhelming amounts of\ntraffic from millions of systems and are not the biggest problem.\n\nWhat isn't clear is who is using the residential proxies; *somebody*\nis paying them to run these attacks on web sites. There is no\nevidence (that I am aware of) that the frontier-model companies are using\nthose networks. If were to turn out that they *are* doing so, though,\nthe increase in global astonishment would barely register. Those companies\nare feeding their models somehow, they are not forthcoming about how they\nget their training data, and they have not distinguished themselves with\ntheir level of respect toward content creators — or toward anybody who\nmight have concerns about their operations.\n\nFor every public model, though, there must be a vast number of undercover models. Many companies are surely trying to build their own; after all, we are reliably informed that AI is going to take over the world and the companies that come out on top of that race will be worth untold amounts of money. There must be shadowy government agencies in many countries working on their own models and groping for training data wherever they can find it. Large-scale criminal organizations (to the extent that they are distinct from governments) probably also want to have their own models. These tools are seen as weapons, and there is an arms race underway. The Internet as a whole is caught in the crossfire.\n\n#### Defending the open Internet\n\nIn response to all of this, web-site operators have been scrambling to\ndefend their sites while minimizing the effect on their actual users. [Anubis](/Articles/1028558/), which attempts to fend off scrapers by\nrequiring a proof of work, is now widespread. Other sites use commercial\nservices, which sometimes make themselves known with a \"prove you are\nhuman\" button. Or sites force users to pick out squares containing\nstreetlights (but only those with LED bulbs), place puzzle pieces, or hum a\nsong while holding down the space bar. Many site features have been placed\nbehind login gates or paywalls. Some sites attempt to actively poison the\ndata sent to scrapers with tools like [iocaine](/Articles/1056953/).\n\nBoth the need to set up and maintain these mechanisms, and the requirement that users cope with them to access a web site, constitute a heavy tax placed on the world as a whole by scrapers and those who pay them.\n\nRecently, LWN was subjected what was, by far, the heaviest scraper attack yet. Thanks to the defenses that have been implemented, the site bore the traffic well enough that most actual readers probably did not even notice. There have been requests to describe the measures we have taken to defend the site; for obvious reasons we do not wish to discuss them in any detail. It is an arms race at this level too.\n\nWhat we *can* say is that we have tried to minimize the impact on real\nreaders as much as possible. We have not gone with tools like Anubis,\npartly because it causes annoying delays for those trying to get to the\nsite, but also partly because it seems inevitable that the scrapers will\neventually find their way around it. Indeed, there are some indications\nthat is already happening. A proof-of-work requirement is not a huge\nobstacle when you have millions of other people's machines to do the work\non.\n\nThere is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.\n\nWe *have* aggressively optimized parts of the site, and found ways to\nminimize expensive operations during times when the site is under attack.\nAnonymous readers may occasionally encounter one of those measures;\nlogged-in users will not. Amusingly, the response time when the site is\nunder attack is often better than during the calm times, when the defensive\nmeasures are dormant. We have learned better than to think that the\nproblem is solved, though; consideration must be given to our next steps\nonce the current measures are no longer effective.\n\nOn July 2, Google [announced](https://cloud.google.com/blog/topics/threat-intelligence/google-continued-disruption-residential-proxy-networks)\nthat it had, in coordination with the US Federal Bureau of Investigation\nand others, taken down a residential-proxy network called \"NetNut\". For\nthe time being, that action would, indeed, seem to have succeeded in\nreducing the level of scraper attacks somewhat. Experience shows, though,\nthat this welcome peace will only last so long. Google takes pains to\npoint out that its Play Store will now check for NetNut-infected apps, but\nall of the major vendors are silent on the topic of why it is so easy to\nput apps with residential-proxy functionality into their app stores.\n\nIt would be good to find a more lasting solution before the entire Internet\nis driven behind defensive walls, and the open network that inspired so\nmuch creativity is lost. The industry that is driving these attacks seems\nentirely at ease with turning independent web sites into smoking craters\nafter having pillaged their contents — an attitude that extends to the\nplanet and its economies as well. Some of us, though, object to that idea\nand will fight against it. Someday, with luck, the world as a whole will\ndecide to hold the companies behind large language models and related\ntechnologies to a minimal ethical standard. Until then, though, this\nbehavior will continue, and we will have no choice but to defend ourselves\nagainst it.", "url": "https://wpnews.pro/news/an-update-on-the-scraper-situation", "canonical_source": "https://lwn.net/SubscriberLink/1080822/990a8a5e2d379085/", "published_at": "2026-07-10 19:38:34+00:00", "updated_at": "2026-07-10 20:05:42.692103+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-ethics", "ai-policy"], "entities": ["LWN", "Google", "IPIDEA", "Bright Data", "Krebs on Security"], "alternates": {"html": "https://wpnews.pro/news/an-update-on-the-scraper-situation", "markdown": "https://wpnews.pro/news/an-update-on-the-scraper-situation.md", "text": "https://wpnews.pro/news/an-update-on-the-scraper-situation.txt", "jsonld": "https://wpnews.pro/news/an-update-on-the-scraper-situation.jsonld"}}