{"slug": "wtf-is-the-ai-training-opt-out-policy-token", "title": "WTF is the AI training opt-out policy token?", "summary": "Anthropic technical lead for standards Kevin Kelley submitted a proposed \"policy token\" for AI training to the Internet Engineering Task Force on Oct. 30, letting publishers add a single line — User-Agent: __usage_ai-training — to robots.txt with an allow or disallow instruction instead of listing each crawler individually. Anthropic, Google, Microsoft and OpenAI have agreed to support the unreviewed draft, which Anthropic says it is rolling out now and OpenAI says it will honor \"in the coming months,\" while SPUR technical lead Alex Springer called the platforms' unilateral backing \"the antithesis\" of IETF consensus-building. The proposal still requires IETF deliberation through its AI Preferences (AIPREF) group before it becomes an official standard.", "body_md": "Join us Dec. 1-3 in Nashville for the Digiday Programmatic Marketing Summit\n\n[REGISTER](https://digiday.com/events/digiday-programmatic-marketing-summit-dec-2026/?source=display&utm-source=display)\n\nThis article is a WTF explainer, in which we break down media and marketing’s most confusing terms. [More from the series →](https://digiday.com/series/wtf/)\n\nSome of the largest AI tech companies — including Anthropic, Google, Microsoft and OpenAI — have agreed to support a new mechanism that lets publishers opt out of having their content used to train AI models, in just a few lines of text.\n\nThe proposed standard, called a “policy token” for AI training, comes as publishers seek more control over how tech companies scrape and use their content. They’ve tried [blocking AI bots](https://digiday.com/media/reuters-and-time-adopt-bot-blocking-whitelists-to-rein-in-ai-crawlers/), pursuing legal action and negotiating licensing deals.\n\nAt first glance, this AI training policy token appears to offer a straightforward fix. In reality, it highlights why the problem is so complicated.\n\n## What is the policy token?\n\nThe idea is that publishers can add a line to their robots.txt file — which websites use to tell web crawlers what content they are allowed to access — to express whether they want their content used for AI training.\n\nHere’s what the policy token looks like, as written in robots.txt: User-Agent: __usage_ai-training\n\nBelow that, publishers can write “disallow” or “allow,” to show which crawlers can scrape and which can’t. In the “allow” line, publishers can also offer certain exceptions, such as letting crawlers scrape a press kit.\n\n## Why does this matter for publishers?\n\nTypically, publishers need to list each crawler individually in robots.txt and give permission (or not) to crawl their sites. The new AI training policy token would allow publishers to do so in one step. Instead of managing separate instructions for every crawler, they can set broad preferences to participating AI and tech companies.\n\n## Who created it?\n\nThe proposal was written by Kevin Kelley, a technical lead for standards at Anthropic. Kelley submitted the proposal to the Internet Engineering Task Force (IETF), an internet standards development organization, on Oct. 30.\n\nTechnically, the standard is not official yet. The IETF launched an effort known as AI Preferences (AIPREF) to develop new standards for website owners to express preferences about how their content gets used for AI. Anthropic, Google, Microsoft and OpenAI have all been involved in the group.\n\nWhile the largest search and AI providers have agreed to support this policy token, it still needs to be deliberated on by the IETF. And the fact that these major companies have jumped to support this draft raises some concerns, said Alex Springer, technical lead at SPUR, a coalition of publishers working to develop AI standards.\n\n“The IETF exists to allow for individuals to work in public to form consensus about the standards and protocols that make the internet work. The way that the platforms have approached this by announcing their unilateral support of a ‘standard’ that is an unreviewed draft is the antithesis of that,” he said. “They appear to have decided to unilaterally wield their collective weight in a concerted effort to force a new de facto standard in place, one that falls far short of what those content owners require.”\n\n## When will this go into effect?\n\nAnthropic says [it is rolling out](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) adoption of the signal now.\n\n[In a blog post,](https://openai.com/index/helping-websites-express-ai-training-preferences/) OpenAI said it would honor the policy token “in the coming months.” It noted that the proposed mechanism would not replace its existing crawler controls.\n\nGoogle [wrote in its own blog post](https://developers.google.com/search/blog/2026/10/ai-training-control) that it aims to support this policy token in 2027. It noted that it builds on the options it already offers publishers, such as Google-Extended, a crawler that [publishers can block](https://digiday.com/media/jargon-buster-the-key-terms-to-know-on-ai-bot-traffic-and-monetization/) in robots.txt to control AI training access, implemented in 2023.\n\nMicrosoft went a bit further, writing [in a blog post](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) that not only would it comply with the AI training policy token but that it would also offer publishers the ability to opt out of AI grounding, or when [crawlers scrape sites](https://digiday.com/media/wtf-is-ai-grounding-licensing-and-why-do-publishers-say-it-matters-over-training-deals/) to find real-time information. These preferences will replace the existing Microsoft preferences “NoArchive” and “NoCache,” for AI training and grounding respectively, starting Feb. 1, 2027.\n\nMeta didn’t go as far as to say it would honor the policy token, but Andy Stone, a Meta company spokesperson, [wrote on X](https://x.com/andymstone/status/2116257949773607300?s=20): “Meta, too, strongly support publishers’ ability to express preferences about how their content is used for foundational generative AI training. We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry.”\n\n## Sounds like this should help curb the AI scraping problem. What’s the catch?\n\nThe proposal may appeal to publishers seeking standardized controls. But, like robots.txt, it offers no way to ensure AI and tech companies comply with publishers’ preferences.\n\nRobots.txt preferences are infamously more of a suggestion than an enforceable doctrine.\n\nIn fact, data suggests that web crawlers rarely even check publishers’ robots.txt files when scraping their sites. Crawlers operated by the four companies backing the shared opt-out token made roughly 1 billion requests over the past 30 days — but just 0.15% of those requests were for robots.txt files, according to data shared with Digiday by AI visibility measurement firm ScalePost, which tracked activity across more than 50 publisher properties.\n\nPaul Bannister, chief strategy officer at Raptive, called the AI training policy token “window dressing.”\n\n“I think this is useless and is marketing, and far, far, far too late… It does nothing and doesn’t solve the problem at all,” he said.\n\nAI tech companies have a lot of reasons to want to appear to appease publishers. They’re fighting a number of [copyright infringement lawsuits](https://digiday.com/media/paywall-violation-eviscerates-fair-use-defense-what-openai-nyt-twist-means-for-publishers/), [pressure from government regulators](https://digiday.com/media/media-briefing-the-transatlantic-push-to-rein-in-stealth-bots-is-gathering-pace/), and [proposed legislation targeting crawlers](https://digiday.com/media/conde-nast-hearst-among-300-media-execs-to-push-federal-bad-bots-bill-on-ai-scraping/). \n\n## So if no one will comply, what’s the point of it?\n\nFour major AI tech companies have said they will honor this AI training policy token, making this one of the first proposed standards with that kind of support.\n\nWhile it remains to be seen if this will be the case in practice, the policy token could be another layer publishers have at their disposal to protect their content from web crawlers, in addition to mechanisms like blocking crawlers at the CDN level or using tools like TollBit.\n\n“Even though it doesn’t keep all the thieves out, you’d still put a lock on your front door because it keeps some of the thieves out… You do lots and lots of things to minimize the chances that your house gets broken into, but you cannot prevent it completely,” Bannister said. “Is it one other arrow to put in your quiver of about 1,000 things you need to do? Sure.”\n\n## If publishers do block AI training crawlers, will they sacrifice their visibility in AI tools?\n\nFor the most part, that’s unclear.\n\nMany publishers have been working hard to improve their AI visibility, and some [are selling GEO products](https://digiday.com/media/media-briefing-publishers-are-turning-geo-from-an-experiment-into-a-business/) to brands to boost their visibility in LLMs. The last thing those publishers want is to jeopardize their mentions and citations in those tools.\n\nOf the four companies backing the policy token, Microsoft was the only one to explicitly address its implications for AI visibility. If a publisher does disallow crawlers for AI grounding, “you will see reduced brand visibility in AI experiences,” reads Microsoft’s blog post.\n\nThis is part of the reason publishers are wary of some of Google’s [AI opt-out mechanisms](https://digiday.com/media/googles-ai-opt-out-leaves-publishers-with-a-choice-they-cant-safely-use/), which let publishers exclude their content from AI-powered search features but could mean sacrificing valuable visibility and traffic they can’t afford to lose.\n\n## What are some other unanswered questions?\n\nThe AI training policy token doesn’t have anything to do with AI inference (or AI usage), which is when an AI model generates responses or performs tasks by retrieving or processing content from publishers’ websites in real time. This has become core to most of the recent AI licensing deals between publishers and tech companies.\n\nAnd even if an AI crawler abides by publishers’ preferences expressed in robots.txt, it doesn’t mean the AI provider won’t be able to get that data elsewhere, [such as from other crawlers](https://digiday.com/media/in-graphic-detail-new-data-shows-publishers-face-growing-ai-bot-third-party-scraper-activity/), Bannister said. \n\n“None of them have said, ‘We won’t buy data from third parties that scrape the content that don’t obey this,’ or ‘We won’t take open source crawls like Common Crawl,’ Bannister said. “They’re going to get all the data anyway, just through a different mechanism, and so it’s completely useless.”\n\nThe draft also appears to reinforce the idea that AI companies can use publishers’ content for training by default, unless publishers explicitly tell them not to, Springer said. This “opt in unless opted out” concept risks legitimizing a system where crawlers can presume they have permission for AI training, rather than needing explicit consent, he argued.\n\n“The platforms are taking the stance that unless a preference to the contrary is expressed then content is fair game for AI training. We should not be normalizing this view and need to continue to work to maintain the concept that the usage of content should be determined by its owners and the result of their direct permission,” Springer said.\n\n### More in Media\n\n#### \n                [As publishers await click data, Google’s AI visibility push complicates the opt-out decision](https://digiday.com/media/as-publishers-await-click-data-googles-ai-visibility-push-complicates-the-opt-out-decision/)            \n\nGoogle is moving on two fronts: working up the U.K’s Competition and Markets Authority-mandated search opt-out publishers have long sought, while enhancing AI Overviews in ways that could make opting out less appealing.\n\n#### \n                [Creators see promise in Instagram’s long-form push, but question whether exposure is enough](https://digiday.com/media/creators-see-promise-in-instagrams-long-form-push-but-question-whether-exposure-is-enough/)             \n\nInstagram wants to pull long-form creators away from YouTube, but is it offering enough? Experts weigh in on the platform’s pivot.\n\n#### \n                [By the numbers: Ad tech’s widening divide, as AppLovin surges and rivals struggle in Q3](https://digiday.com/media/by-the-numbers-ad-techs-widening-divide-as-applovin-surges-and-rivals-struggle-in-q3/)            \n\nAd tech’s winners and losers emerge as CTV growth masks open-web struggles.", "url": "https://wpnews.pro/news/wtf-is-the-ai-training-opt-out-policy-token", "canonical_source": "https://digiday.com/media/wtf-is-the-ai-training-opt-out-policy-token/?utm_campaign=digidaydis&utm_medium=rss&utm_source=general-rss", "published_at": "2026-11-09 05:01:00+00:00", "updated_at": "2026-11-09 05:24:41.348259+00:00", "lang": "en", "topics": ["ai-policy", "ai-crawlers", "agent-protocols", "artificial-intelligence"], "entities": ["Anthropic", "Kevin Kelley", "Internet Engineering Task Force", "Google", "Microsoft", "OpenAI", "SPUR", "Alex Springer"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/wtf-is-the-ai-training-opt-out-policy-token", "markdown": "https://wpnews.pro/news/wtf-is-the-ai-training-opt-out-policy-token.md", "text": "https://wpnews.pro/news/wtf-is-the-ai-training-opt-out-policy-token.txt", "jsonld": "https://wpnews.pro/news/wtf-is-the-ai-training-opt-out-policy-token.jsonld"}}