On Friday, four of the largest AI companies announced a new robots.txt standard that would instruct all of their crawlers to not use the site’s content for AI training. Theoretically, this would end the requirement for sites to block each bot individually and, instead, would allow for one block to apply to all of the major AI companies.
The companies that collaborated on the new standard are Anthropic, OpenAI, Google, and Microsoft. It creates a new token, __usage_ai-training, which can be used to block or allow all major AI crawlers with just two lines of code.
But while the new standard is much simpler, it doesn’t give publishers and rights holders any new tools to control how their content is used. The problems that limited the effectiveness of such a standard remain in place.
Because of that, the new standard isn’t likely to be a game-changer for webmasters and rights holders. If it did, I highly doubt that the AI companies would be offering it in the first place.
Understanding the New Standard
To understand what the AI companies did, we have to first look at the Robots Exclusion Protocol, which is better known as robots.txt.
Robots.txt is a plain text file that is placed in a website’s root directory. For example, you can see this site’s robots.txt here. It is a way for webmasters to communicate with crawlers and other bots and state their preferences for what they want and do not want to be crawled.
Though most “good” crawlers respect robots.txt, the standard doesn’t actually block bots from accessing parts of a site. Because of that, there is an entire ecosystem of tools and products that attempt to block unwanted bots from accessing content.
Still, the robots.txt standard allows webmasters to single out specific bots by their user-agent. For example, if you wanted to allow Google search crawlers to access your site, you would allow “Googlebot” in your robots.txt file. If you wanted to block Google’s AI crawlers, you would block access to “Google-Extended.”
However, as more and more companies are entering the AI space, the list of user agents one would have to block has grown comically large. This new standard aims to simplify that process. Rather than listing each user agent individually, webmasters can simply use the new token __usage_ai-training to block all AI crawlers from Google, Microsoft, OpenAI, and Anthropic.
It is a nice idea, and it would be especially great if it became the de facto standard for all AI companies, not just the major ones.
However, this announcement papers over some big problems. The biggest is that robots.txt has been a dying standard for a long time, and that, for most sites, is akin to closing the barn door after the horses have already escaped.
The Problem with the Standard
Back in June, I wrote an article saying that, for the time being, I was giving up on blocking AI bots. That remains the case today.
The reason is simple: blocking AI crawlers, regardless of approach, has failed to prevent my work from ending up in AI training datasets. Whether it was obtained directly from this site, from a pirate dataset, an archive, or some other third party, all of the AI bots have had no trouble accessing my work.
If using robots.txt to block these bots directly didn’t work previously, then what reason do I have to believe that this new standard will work any better? It’s simple: I don’t.
But even if it is more effective, it will only impact future articles. I have more than 20 years of work on this site, and I have to assume that all of it has been used by AI companies, both large and small. That cannot be undone. Everything I ever wrote was already in AI datasets before anyone was even thinking about blocking AI bots.
So while this new standard could be somewhat useful if it becomes more widely adopted, it doesn’t address any of the actual issues that most webmasters and rights holders have when it comes to how AI has used their content.
But if AI companies actually did address those issues, it’s unlikely that they would function as AI companies. AI as it exists today requires massive amounts of human-created content, almost all of it gathered without direct consent.
Bottom Line
While I’m sure it took a great deal of work to get these four rival companies to agree on the new standard, it’s important to put what they achieved into perspective. All that they have done is create a simplified way for webmasters to indicate that they do not want their content to be used for AI training. The standard it is built upon, robots.txt, is fully optional, and webmasters could already signal their intent before this announcement.
Webmasters and creators have no new powers, just a slightly easier way to flex the very limited control that they have always had. However, that control has already proven to be completely ineffective.
In the end, this strikes me more as a feel-good announcement for the AI companies than an actual boon for webmasters. It’s the appearance of doing something without actually doing anything.
But then again, AI companies can’t actually do anything. Without scrapping everything they’ve done and starting over from the ground up with a permission-first model, there is no way to truly address these issues. With billions of dollars on the line, it’s just not going to happen.
Instead, they’ll continue to nibble at the edges of the permission problem, while ignoring the elephant in the room.
Want to Reuse or Republish this Content?
If you want to feature this article in your site, classroom or elsewhere, just let us know! We usually grant permission within 24 hours.