Join us Dec. 1-3 in Nashville for the Digiday Programmatic Marketing Summit
REGISTER This article is a WTF explainer, in which we break down media and marketing’s most confusing terms. More from the series →
Some of the largest AI tech companies — including Anthropic, Google, Microsoft and OpenAI — have agreed to support a new mechanism that lets publishers opt out of having their content used to train AI models, in just a few lines of text.
The proposed standard, called a “policy token” for AI training, comes as publishers seek more control over how tech companies scrape and use their content. They’ve tried blocking AI bots, pursuing legal action and negotiating licensing deals.
At first glance, this AI training policy token appears to offer a straightforward fix. In reality, it highlights why the problem is so complicated.
What is the policy token? #
The idea is that publishers can add a line to their robots.txt file — which websites use to tell web crawlers what content they are allowed to access — to express whether they want their content used for AI training.
Here’s what the policy token looks like, as written in robots.txt: User-Agent: __usage_ai-training
Below that, publishers can write “disallow” or “allow,” to show which crawlers can scrape and which can’t. In the “allow” line, publishers can also offer certain exceptions, such as letting crawlers scrape a press kit.
Why does this matter for publishers? #
Typically, publishers need to list each crawler individually in robots.txt and give permission (or not) to crawl their sites. The new AI training policy token would allow publishers to do so in one step. Instead of managing separate instructions for every crawler, they can set broad preferences to participating AI and tech companies.
Who created it? #
The proposal was written by Kevin Kelley, a technical lead for standards at Anthropic. Kelley submitted the proposal to the Internet Engineering Task Force (IETF), an internet standards development organization, on Oct. 30.
Technically, the standard is not official yet. The IETF launched an effort known as AI Preferences (AIPREF) to develop new standards for website owners to express preferences about how their content gets used for AI. Anthropic, Google, Microsoft and OpenAI have all been involved in the group.
While the largest search and AI providers have agreed to support this policy token, it still needs to be deliberated on by the IETF. And the fact that these major companies have jumped to support this draft raises some concerns, said Alex Springer, technical lead at SPUR, a coalition of publishers working to develop AI standards. “The IETF exists to allow for individuals to work in public to form consensus about the standards and protocols that make the internet work. The way that the platforms have approached this by announcing their unilateral support of a ‘standard’ that is an unreviewed draft is the antithesis of that,” he said. “They appear to have decided to unilaterally wield their collective weight in a concerted effort to force a new de facto standard in place, one that falls far short of what those content owners require.”
When will this go into effect? #
Anthropic says it is rolling out adoption of the signal now. In a blog post, OpenAI said it would honor the policy token “in the coming months.” It noted that the proposed mechanism would not replace its existing crawler controls.
Google wrote in its own blog post that it aims to support this policy token in 2027. It noted that it builds on the options it already offers publishers, such as Google-Extended, a crawler that publishers can block in robots.txt to control AI training access, implemented in 2023.
Microsoft went a bit further, writing in a blog post that not only would it comply with the AI training policy token but that it would also offer publishers the ability to opt out of AI grounding, or when crawlers scrape sites to find real-time information. These preferences will replace the existing Microsoft preferences “NoArchive” and “NoCache,” for AI training and grounding respectively, starting Feb. 1, 2027.
Meta didn’t go as far as to say it would honor the policy token, but Andy Stone, a Meta company spokesperson, wrote on X: “Meta, too, strongly support publishers’ ability to express preferences about how their content is used for foundational generative AI training. We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry.”
Sounds like this should help curb the AI scraping problem. What’s the catch? #
The proposal may appeal to publishers seeking standardized controls. But, like robots.txt, it offers no way to ensure AI and tech companies comply with publishers’ preferences.
Robots.txt preferences are infamously more of a suggestion than an enforceable doctrine.
In fact, data suggests that web crawlers rarely even check publishers’ robots.txt files when scraping their sites. Crawlers operated by the four companies backing the shared opt-out token made roughly 1 billion requests over the past 30 days — but just 0.15% of those requests were for robots.txt files, according to data shared with Digiday by AI visibility measurement firm ScalePost, which tracked activity across more than 50 publisher properties.
Paul Bannister, chief strategy officer at Raptive, called the AI training policy token “window dressing.”
“I think this is useless and is marketing, and far, far, far too late… It does nothing and doesn’t solve the problem at all,” he said.
AI tech companies have a lot of reasons to want to appear to appease publishers. They’re fighting a number of copyright infringement lawsuits, pressure from government regulators, and proposed legislation targeting crawlers.
So if no one will comply, what’s the point of it? #
Four major AI tech companies have said they will honor this AI training policy token, making this one of the first proposed standards with that kind of support.
While it remains to be seen if this will be the case in practice, the policy token could be another layer publishers have at their disposal to protect their content from web crawlers, in addition to mechanisms like blocking crawlers at the CDN level or using tools like TollBit. “Even though it doesn’t keep all the thieves out, you’d still put a lock on your front door because it keeps some of the thieves out… You do lots and lots of things to minimize the chances that your house gets broken into, but you cannot prevent it completely,” Bannister said. “Is it one other arrow to put in your quiver of about 1,000 things you need to do? Sure.”
If publishers do block AI training crawlers, will they sacrifice their visibility in AI tools? #
For the most part, that’s unclear. Many publishers have been working hard to improve their AI visibility, and some are selling GEO products to brands to boost their visibility in LLMs. The last thing those publishers want is to jeopardize their mentions and citations in those tools.
Of the four companies backing the policy token, Microsoft was the only one to explicitly address its implications for AI visibility. If a publisher does disallow crawlers for AI grounding, “you will see reduced brand visibility in AI experiences,” reads Microsoft’s blog post.
This is part of the reason publishers are wary of some of Google’s AI opt-out mechanisms, which let publishers exclude their content from AI-powered search features but could mean sacrificing valuable visibility and traffic they can’t afford to lose.
What are some other unanswered questions? #
The AI training policy token doesn’t have anything to do with AI inference (or AI usage), which is when an AI model generates responses or performs tasks by retrieving or processing content from publishers’ websites in real time. This has become core to most of the recent AI licensing deals between publishers and tech companies.
And even if an AI crawler abides by publishers’ preferences expressed in robots.txt, it doesn’t mean the AI provider won’t be able to get that data elsewhere, such as from other crawlers, Bannister said.
“None of them have said, ‘We won’t buy data from third parties that scrape the content that don’t obey this,’ or ‘We won’t take open source crawls like Common Crawl,’ Bannister said. “They’re going to get all the data anyway, just through a different mechanism, and so it’s completely useless.”
The draft also appears to reinforce the idea that AI companies can use publishers’ content for training by default, unless publishers explicitly tell them not to, Springer said. This “opt in unless opted out” concept risks legitimizing a system where crawlers can presume they have permission for AI training, rather than needing explicit consent, he argued.
“The platforms are taking the stance that unless a preference to the contrary is expressed then content is fair game for AI training. We should not be normalizing this view and need to continue to work to maintain the concept that the usage of content should be determined by its owners and the result of their direct permission,” Springer said.
More in Media
[As publishers await click data, Google’s AI visibility push complicates the opt-out decision](https://digiday.com/media/as-publishers-await-click-data-googles-ai-visibility-push-complicates-the-opt-out-decision/)
Google is moving on two fronts: working up the U.K’s Competition and Markets Authority-mandated search opt-out publishers have long sought, while enhancing AI Overviews in ways that could make opting out less appealing.
[Creators see promise in Instagram’s long-form push, but question whether exposure is enough](https://digiday.com/media/creators-see-promise-in-instagrams-long-form-push-but-question-whether-exposure-is-enough/)
Instagram wants to pull long-form creators away from YouTube, but is it offering enough? Experts weigh in on the platform’s pivot.
[By the numbers: Ad tech’s widening divide, as AppLovin surges and rivals struggle in Q3](https://digiday.com/media/by-the-numbers-ad-techs-widening-divide-as-applovin-surges-and-rivals-struggle-in-q3/)
Ad tech’s winners and losers emerge as CTV growth masks open-web struggles.