![4 AI Giants Back One AI Opt-Out Token [2026]](https://shattered.io/wp-content/uploads/2026/11/ai-training-opt-out-robots-txt-token-openai-google-microsoft-anthropic-2026-gen.webp)
Four of the biggest names in AI agreed on October 30, 2026 to honor a single robots.txt signal for AI training opt-outs. OpenAI, Google, Microsoft and Anthropic will each read the same __usage_ai-training token, according to reports from PPC Land, ShiftDelete.net and others. The Indian Express and several other outlets have framed it as a way for publishers to stop maintaining a separate rule for every AI company.
The idea is simple. A publisher adds one group to robots.txt, and every participating company treats it as a request not to train generative models on the covered pages. That replaces a growing pile of crawler-specific entries with one line of intent. The catch is that nobody has shipped it across the board yet, it is not an official IETF standard, and it covers training only.
This matters because publishers have spent two years watching referral traffic erode while AI products reuse their work. Our earlier look at Reddit’s complaint about Google’s AI Mode showed how raw that tension has become. A shared opt-out will not fix the traffic problem. It does make the consent side of the ledger easier to manage, and that is why this story has spread so fast.
Don't miss new tech stories on Google
Add Shattered once in the Google app and our stories show up in your news suggestions.
What the Four Companies Announced on October 30 #
The announcement came from four companies at once, which is unusual in an industry that rarely agrees on anything. According to the reporting, OpenAI, Google, Microsoft and Anthropic committed to honoring one robots.txt policy token that tells AI developers not to use a site’s content for model training. The shared token replaces individual crawler rules with a single group, so a publisher no longer has to list GPTBot, Google-Extended, ClaudeBot and every future bot by name.
Google’s own framing, as relayed by PPC Land, was that site owners would stop managing separate instructions for each AI developer. OpenAI described the work as a voluntary effort. That wording is worth keeping in mind, because it sets expectations. Nothing in the announcement creates a legal duty, and no regulator is involved.
The commitments also differ in timing. Anthropic has already started implementing the signal, per the reports. Microsoft plans to begin on February 1, 2027. Google is aiming for 2027, with PPC Land putting it in early 2027. OpenAI says only that it will arrive in the coming months. So on the day of the announcement, the practical effect for a publisher was close to zero, and the real change lands over the next two quarters.
Two details separate this from a typical press-release pledge. First, the token is tied to a written definition of AI training from the IETF working group, so the companies are pointing at a shared vocabulary instead of their own wording. Second, the author of the underlying Internet-Draft works at one of the participants, which suggests the text was negotiated in the open before the companies signed on.
How the __usage_ai-training Token Works #
The mechanism borrows the shape of robots.txt that every webmaster already knows. The token appears in a User-Agent line, followed by Allow or Disallow rules. A crawler that honors it reads the group, applies the usual path rules, and treats a Disallow as a request not to train on that content. Because it reuses existing syntax, the learning curve for site owners is short.
User-Agent: __usage_ai-training
Disallow: /
That two-line group blocks AI training across the whole site. A publisher who wants to protect only premium or archive material can scope the Disallow to specific paths instead. Per-path control is part of the design, and path matching follows the precedence rules from RFC 9309, the formal robots.txt specification, according to PPC Land’s reading of the draft.
Exact Syntax and Matching Rules
Three rules trip people up. A wildcard group such as User-Agent: * does not match the policy token, so a blanket rule written years ago will not opt anyone out. A site with no stated preference is treated as allowing AI training by default. And private licensing agreements between a publisher and an AI company can override whatever the token says.
The draft’s author, Kevin Kelley, made the default point sharply in the text. His position, in paraphrase, is that a robots.txt file that does not use this mechanism says nothing about training preferences at all. That is true of every robots.txt file published today. Old blocks aimed at search crawlers should not be read as AI consent or refusal either way.
Who Wrote It and Where It Sits in the IETF #
The token comes from the IETF’s AI Preferences working group, known as AIPREF, but the document that defines it is not a working group product. It is an individual Internet-Draft by Kevin Kelley, a standards lead at Anthropic, titled draft-kelley-aipref-training-preference-signal-00. PPC Land reports that the draft expires on May 3, 2027 and carries no formal IETF standing.
The token takes its meaning from the AI Training definition in draft-ietf-aipref-vocab-08, the working group’s vocabulary document. PPC Land credits that draft to Paul Keller and Martin Thomson and dates it to mid-September 2026. A companion draft, draft-ietf-aipref-attach-05 by Gary Illyes and Martin Thomson, covers how preferences attach to content through robots.txt and HTTP headers. It dates to August 2026.
The explainx.ai analysis adds a caution that deserves repeating. The vocabulary draft itself notes the lack of working-group consensus, and the attachment draft has not gone to review by the Internet Engineering Steering Group. In plain terms, the plumbing is still being argued over while four companies agree to use a simplified slice of it.
Kelley’s draft lists 46 acknowledged contributors, per PPC Land. They include staff from Google, Microsoft, Meta, OpenAI, Amazon and Cloudflare, plus people from News Corp, the Financial Times, the BBC, Elsevier and NBCUniversal. The list also names the Electronic Frontier Foundation, the Center for Democracy and Technology, Creative Commons and the Internet Archive. Being thanked in a draft is not an endorsement, so the presence of a company on that list does not mean it has committed to the token.
Company-by-Company Timeline #
The table below sets out who has committed to what, based on the reporting from PPC Land and the explainx.ai summary of The Verge’s coverage. Treat dates as the companies’ stated plans, since none had shipped broad support at the time of writing.
| Company | Stated timing | Reported detail |
|---|---|---|
| Anthropic | Already started | Kelley said implementation is under way |
| Microsoft | February 1, 2027 | Adds page-level AI Training meta tag and header option |
| 2027 (early 2027 per PPC Land) | Treats the shared token as a Google-Extended disallow | |
| OpenAI | Coming months | No date given, scoped to foundational training |
| Meta | Not committed | Support signaled in some coverage, status unconfirmed |
| Apple | Not committed | Applebot-Extended exists, token not mentioned |
| Amazon | Not committed | Listed among contributors, no pledge reported |
The absence of Meta, Apple and Amazon from the commitment list is the first gap to watch. Those three train large models and run crawlers of their own. Until they join, a publisher who adds the token blocks four of the leading developers and leaves others untouched, so existing per-crawler rules stay necessary.
What the Token Does Not Cover #
Headlines about a single opt-out invite a bigger reading than the text supports. The token addresses training, meaning the modification of a generative model’s parameters. It says nothing about whether a crawler may fetch a page, whether a search engine may rank it, or whether an assistant may quote it live while answering a question.
| Use of content | Covered by the token? | Reported basis |
|---|---|---|
| Training generative AI models | Yes | Definition in draft-ietf-aipref-vocab-08 |
| Per-path control | Yes | Standard robots.txt path rules |
| Gemini grounding | Yes, via Google-Extended | Google treats the token as a Google-Extended disallow |
| Crawling itself | No | Existing crawlers keep running |
| Search rankings | No | Googlebot and Bingbot rules unchanged |
| Retrieval at OpenAI or Anthropic | No | Scoped to training only |
| AI Overviews and AI Mode | No | Separate Search Console control since July 2026 |
The retrieval gap is the one publishers will feel most. A site can block training and still see its articles summarized in an AI answer, because retrieval-time use falls outside the token. Anyone expecting one line in robots.txt to end all AI reuse of their work will be disappointed.
Google’s Reading: Google-Extended Gets a Shared Door #
Google’s implementation is the most consequential, because Google controls the largest crawl footprint on the web. According to PPC Land, Google will treat a shared-token opt-out as equivalent to a Google-Extended disallow. That control already governs training for Gemini models, and it also reaches Gemini grounding. An explicit Google-Extended allow can override the shared token’s block, which means a publisher with conflicting lines gets the more specific one.
Google also drew a boundary around its search AI. The shared token does not affect AI Overviews or AI Mode, which a separate Search Console control has governed since July 2026. Google cited the EU’s General-Purpose AI Code of Practice as part of the rationale. Publishers who want out of AI Mode still have to use that other control.
The split fits a pattern we have tracked. Google keeps widening what Gemini can reach, as in the connected-apps platform that opens Gemini and Search to 50+ apps, while carving out narrow opt-out controls for specific uses. The token simplifies one of those controls. It does not change the bigger direction of travel.
Microsoft’s Extra Controls and the Grounding Tradeoff #
Microsoft went further than the other three on tooling. It plans a separate page-level AI Training meta tag and HTTP header option, so publishers can set preferences per page without touching robots.txt. PPC Land also reports a three-tier grounding preference with Allow as the default, Disallow, and a DisallowExtended setting that applies to Microsoft products only and excludes third-party grounding.
Temporary grounding preferences are due to open in Bing Webmaster Tools on December 15, 2026, per the same report. PPC Land further says Microsoft will stop reading the older NoArchive and NoCache directives after March 1, 2027. If that detail holds, publishers who rely on those tags for AI-related effects need to migrate before then.
Microsoft also attached a plain warning. PPC Land reports that the company says neither choice guarantees inclusion, traffic or payment. That tells publishers the company is not promising a trade of data for exposure. Opting out may cost visibility inside AI experiences, and opting in does not buy a payout.
Why Publishers Wanted This #
The pressure comes from arithmetic. Every new AI developer launches with its own crawler name and its own opt-out page. A publisher with a legal team can track a dozen of them. A small outlet with one developer cannot. The result has been blocklists copied from forums and rules that go stale the week a new bot appears.
The Per-Crawler Treadmill
Per-crawler rules also fail in a quieter way. A publisher who blocks GPTBot today has no say over a model that launches next spring under a name nobody has heard. A token tied to a purpose, training, survives new brands because the rule describes what the content is used for rather than who is asking. That purpose-based design is the real innovation here, more than the specific string.
The same dynamic is hitting platforms. Reddit’s push to force sign-ins was a response to AI-driven search changes, and we covered the fallout when search referrals dropped by more than 1 million a week. Publishers that depend on search traffic are weighing consent against exposure in a market where both are shrinking.
The Search Visibility Catch #
Both Google and Microsoft warn that opting out of AI uses can reduce how often a site appears in AI experiences, according to the reports. That is a soft threat, and publishers should read it as one. The token covers training, but Google ties its interpretation to Gemini grounding, and Microsoft’s grounding tiers move content in or out of its answers.
The money behind that warning is large. Search remains the profit engine for the companies at the table, and AI answers are where they are steering it. Our breakdown of Google’s AI Mode monetization and its $63.3B search revenue shows why Google wants publisher content flowing into those answers. A publisher that opts out is making a bet that long-term control beats short-term reach.
For news sites with strong direct audiences, the bet looks easy. For niche sites that live on long-tail search, it is harder. The token gives both groups the same switch, and it leaves the judgment about visibility where it was before, with the publisher.
The Voluntary Compliance and Enforcement Gap #
Robots.txt has always worked on trust. A crawler can ignore it with no technical penalty, and the new token inherits that weakness. The explainx.ai review lists the limits bluntly. Compliance is voluntary, changes affect future crawls only, cached robots.txt files can stay valid for up to 24 hours, and there is no public audit trail showing a company honored the signal.
Not retroactive is the clause with the most bite. If a model trained on a site’s archive before the token existed, flipping the switch now removes nothing from that model. Publishers who care about past use need to pursue it through contracts or courts, not robots.txt. Anthropic has taken a visible policy stance on court oversight of risky models, which we covered in its letter urging courts to block risky AI, but the opt-out token is a separate, technical matter with no court role.
There is also the problem of companies outside the pledge. Four developers agreeing to a norm does little against scrapers that never intended to follow one. Disputes over how rivals gather training data have already reached the news, as in the reported distillation of GPT-6 Astra by DeepSeek and Moonshot. A polite request in a text file does not stop that kind of extraction. Hard blocks still need rate limits, authentication or paywalls.
Shared Token vs Per-Crawler Rules vs Licensing Efforts #
Publishers now have three broad ways to express AI preferences. They are not exclusive, and most sites will end up using more than one. The comparison below is qualitative and based only on what the reporting describes.
| Approach | Effort for publisher | Reach | Main weakness |
|---|---|---|---|
Shared __usage_ai-training token |
Low, one group | OpenAI, Google, Microsoft, Anthropic once shipped | Training only, voluntary, not yet shipped everywhere |
| Per-crawler rules (GPTBot, Google-Extended, ClaudeBot) | High, grows with each new bot | Only the bots you list | Misses new crawlers |
| Licensing-focused efforts such as Really Simple Licensing | Medium, needs terms and tooling | Depends on buyer adoption | Needs AI firms to pay or comply |
| Hard technical blocks (rate limits, login walls) | High | Any scraper | Hurts real readers and search |
The token does not replace the other columns. It removes the repetition from the first one. A publisher can keep crawler-specific rules as a fallback while the shared signal rolls out, then trim them once each company confirms support. Licensing deals keep their own logic, and the draft says private agreements can override the signal anyway.
The model side of the market sees this differently. Open-weight developers publish weights and often train on broad web data, and they sit outside most robots.txt negotiation. Our comparison of six open-weight model families shows how varied those projects are. None of the four pledging companies speaks for them.
Historical Context: From Search Etiquette to AI Preferences #
Robots.txt began as a courtesy between webmasters and search crawlers and stayed informal for decades. The IETF published it as RFC 9309 only after the web had run on it for a long time. That history explains both the strength and the limit of the new token. Everyone knows the format, and nobody can force a crawler to follow it.
The AI era strained that courtesy. Companies introduced purpose-specific controls such as Google-Extended and Applebot-Extended to separate AI training from search indexing. Each was a patch, and each created its own list to maintain. The IETF formed the AI Preferences working group to build a shared vocabulary so that the patches could converge.
The October 30 announcement is the first time major developers have publicly agreed to read one signal from that vocabulary. Even so, the path is unfinished. The individual draft expires in May 2027, the attachment draft carries its own expiry, and the working group still has to reach consensus. The companies are moving faster than the standards process, and that gap is where surprises tend to appear.
What Publishers Should Do This Week #
Start with an audit. List every AI crawler rule currently in your robots.txt and note which ones you added by copying a template. The existing rules stay valid under the announcement, so nothing needs to be deleted. Then decide, section by section, what you want to protect. Premium archives, paywalled analysis and original reporting usually get stricter treatment than public press releases.
Add the shared group once you have decided. A path-scoped version looks like this.
User-Agent: __usage_ai-training
Disallow: /premium/
Disallow: /archive/
Allow: /
Be careful with one point. The explainx.ai review notes that OpenAI has not published its own syntax, and the IETF attachment draft uses a different form, a Content-Usage line in robots.txt. Until each company documents what it reads, the token in the Kelley draft is the one the four companies are reported to share. Re-check vendor documentation before you rely on it.
Finally, treat the signal as a request. Keep server logs, watch for the named crawlers, and use rate limits or authentication wherever a leak would cost real money. For wider context on how models and data rights intersect, our AI and machine learning coverage tracks each new development as it lands.
Five Predictions and the Market Impact #
These are analysis, not reported fact. They reflect the pattern of the announcement and the stated timelines.
- OpenAI will publish syntax before Microsoft’s February 1, 2027 start. Three of four companies have given dates or started work, and OpenAI is the one still without a stated format.
- Meta will be pressed to commit within months. It is a named contributor to the draft, and publishers will use its absence as leverage.
- Adoption of the token in robots.txt files will stay low until Google ships. Most sites will wait for the largest crawler to confirm support.
- The IETF drafts will change before they settle. The vocabulary draft notes missing consensus, so wording around AI Training may shift.
- Pressure will move from training to retrieval. Once training opt-outs are easy, publishers will focus on the uncovered gap, where AI answers quote live pages.
The common thread is that the easy part is done. Agreeing on a label for training was achievable because the companies already offer similar controls. The harder fights over retrieval, payment and past use are still ahead.
The immediate market effect is small. No company changed its crawling behavior on October 30, and the dates that matter fall in 2027. The strategic effect is larger. By pointing to a shared definition, the four companies have accepted that training consent is a publisher decision with a standard form, which makes it harder to argue later that no clear signal existed.
For publishers, the change cuts administrative work and gives a clean statement of intent. For AI developers, it creates a record of what each site asked for, and that record can matter in future disputes. The takeaway: adopt the token once your target vendors document support, keep your per-crawler rules until then, and plan separately for retrieval, because that is the fight the token leaves open.
Frequently Asked Questions #
What is the AI training opt-out token?
It is a robots.txt policy token called __usage_ai-training. A publisher lists it in a User-Agent line with Disallow rules to ask AI developers not to train generative models on the covered pages. OpenAI, Google, Microsoft and Anthropic announced on October 30, 2026 that they will honor it.
Is it an official IETF standard?
No. The token is defined in an individual Internet-Draft by Kevin Kelley of Anthropic, draft-kelley-aipref-training-preference-signal-00, which reportedly expires on May 3, 2027. It builds on the IETF AIPREF working group’s vocabulary, but the working group has not finished its standard.
When will each company start honoring it?
Anthropic has already started implementing it. Microsoft plans to begin on February 1, 2027, and Google is aiming for 2027. OpenAI says it will honor the signal in the coming months without naming a date.
Does the token stop AI companies from crawling my site?
No. It covers training only. Existing crawlers such as GPTBot, Google-Extended and ClaudeBot keep running, and the token does not change search crawling or rankings. Retrieval-time use at OpenAI and Anthropic also falls outside it.
Will opting out hurt my search visibility?
Google and Microsoft both warn that opting out of AI uses can reduce how often a site appears in AI experiences. Googlebot and Bingbot rules are unchanged, so standard search indexing is not affected by the token itself, according to the reports.
Does the token remove my content from models already trained?
No. The signal affects future crawls only. Content collected before a company honors the token is not withdrawn, so past use has to be handled through contracts or legal action.
Do I need to delete my existing AI crawler rules?
No. Existing per-crawler rules remain valid and unaffected. Keep them until each company confirms it reads the shared token, and add the new group alongside them.
Which AI companies have not committed?
Reports say Meta, Apple and Amazon have not made commitments to the token. Some coverage says Meta signaled support, but that status is unconfirmed. Smaller developers and open-weight projects also sit outside the pledge.