Internet Archive to New York: Don’t Kill the Good Bots in the Fight Against Bad Bots

The Internet Archive, along with the Electronic Frontier Foundation and 18 other civil liberties organizations, urged New York Governor Kathy Hochul to veto the Stealth Crawler Prohibition Act, warning that the law would harm legitimate web crawlers used by archivists, researchers, and journalists while failing to address excessive AI scraping. The proposed law would require all automated access tools to disclose their identity and purpose, and allow news publishers to demand disclosure without evidence of wrongdoing, potentially enabling retaliation against beneficial bots.

We’re joining with EFF and other civil society groups asking the Governor of New York to veto the Stealth Crawler Prohibition Act . Here’s why. News publishers have a real problem: AI companies are sending aggressive, anonymous bots to scrape websites at a scale that strains servers https://www.washingtonpost.com/technology/2025/06/11/tollbit-ai-bot-retrieval/ and drives up infrastructure costs. It’s happening not just to publishers, but also to Wikipedia https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ , libraries https://journal.code4lib.org/articles/18489 and to the Internet Archive https://blog.archive.org/2023/05/29/let-us-serve-you-but-dont-bring-us-down/ itself. We all have a stake in finding workable solutions. But not all bots are bad. Long before AI, automated tools, which many people colloquially call “bots,” such as web crawlers have helped preserve the web, support research, improve security, and power investigative journalism. Libraries use them to archive disappearing websites https://blog.archive.org/2025/09/24/sonomawildfires/ . Researchers use them to study the web https://arxiv.org/abs/2510.10315 and test security. Journalists use them https://pulitzercenter.org/resource/how-non-coding-journalists-can-build-web-scrapers-ai-examples-and-prompts-included to uncover how powerful companies and institutions operate online. These are some of the good bots that power the online infrastructure that helps keep us safe and hold the powerful accountable. As EFF explains https://www.eff.org/deeplinks/2026/07/stealth-crawlers-are-not-threat-open-web-bills-targeting-them-would-be , New York’s Stealth Crawler Prohibition Act risks killing off the good bots along with the bad in an attempt to prohibit abusive AI scraping. The law, now sitting on Governor Hochul’s desk, would require every operator of an automated access tool to disclose their identity, affiliation, purpose, as well as all potential uses of the public news content they access. It would also let any news publisher demand that any anonymous tool unmask itself, and pursue court orders to force that disclosure, without first showing any evidence of wrongdoing. This mandatory identification would allow websites to block or engage in other forms of retaliation against archivists, journalists, or researchers that use these tools for the public good. Further, because the law only applies to news websites, it would do nothing to help Wikipedia and other knowledge repositories that are burdened by aggressive crawlers. The problem isn’t anonymous bots. The problem is excessive, harmful scraping. We need targeted solutions for abusive AI practices, while actively protecting the rights of libraries, researchers, journalists, and readers alike. That’s why the Internet Archive has joined https://www.eff.org/deeplinks/2026/08/eff-joins-18-civil-rights-organizations-calling-governor-hochul-reject-stealth EFF and other civil liberties organizations in asking Governor Hochul to veto the Stealth Crawler Prohibition Act. It’s important to stop bad actors from straining news sites, but a law written for this purpose shouldn’t hand corporations a tool to dismantle crucial civic infrastructure. Let’s work together to solve the real problem without undermining the tools that preserve knowledge, enable journalism, and support democracy. Read the full letter here https://www.eff.org/document/ny-s9934a-stealth-crawlers-request-veto .