We’re joining with EFF and other civil society groups asking the Governor of New York to veto the Stealth Crawler Prohibition Act. Here’s why.
News publishers have a real problem: AI companies are sending aggressive, anonymous bots to scrape websites at a scale that strains servers and drives up infrastructure costs. It’s happening not just to publishers, but also to Wikipedia, libraries and to the Internet Archive itself. We all have a stake in finding workable solutions.
But not all bots are bad. Long before AI, automated tools, which many people colloquially call “bots,” such as web crawlers have helped preserve the web, support research, improve security, and power investigative journalism. Libraries use them to archive disappearing websites. Researchers use them to study the web and test security. Journalists use them to uncover how powerful companies and institutions operate online. These are some of the good bots that power the online infrastructure that helps keep us safe and hold the powerful accountable.
As EFF explains, New York’s Stealth Crawler Prohibition Act risks killing off the good bots along with the bad in an attempt to prohibit abusive AI scraping. The law, now sitting on Governor Hochul’s desk, would require every operator of an automated access tool to disclose their identity, affiliation, purpose, as well as all potential uses of the public news content they access. It would also let any news publisher demand that any anonymous tool unmask itself, and pursue court orders to force that disclosure, without first showing any evidence of wrongdoing. This mandatory identification would allow websites to block or engage in other forms of retaliation against archivists, journalists, or researchers that use these tools for the public good. Further, because the law only applies to news websites, it would do nothing to help Wikipedia and other knowledge repositories that are burdened by aggressive crawlers.
The problem isn’t anonymous bots. The problem is excessive, harmful scraping. We need targeted solutions for abusive AI practices, while actively protecting the rights of libraries, researchers, journalists, and readers alike. That’s why the Internet Archive has joined EFF and other civil liberties organizations in asking Governor Hochul to veto the Stealth Crawler Prohibition Act. It’s important to stop bad actors from straining news sites, but a law written for this purpose shouldn’t hand corporations a tool to dismantle crucial civic infrastructure. Let’s work together to solve the real problem without undermining the tools that preserve knowledge, enable journalism, and support democracy. Read the full letter here.