OpenAI, Google, Microsoft and Anthropic have all said they will honour the shared AI training signal in robots.txt. Image: OpenAI and Microsoft logos via LobeHub Icons (MIT), Google and Anthropic logos via Simple Icons (CC0); trademarks of their owners
OpenAI, Google, Microsoft and Anthropic have agreed to honour a single new line in a website’s robots.txt file that says “don’t train your AI on this”, so site owners will no longer have to block each company’s crawler one by one. OpenAI announced on Friday that it will respect the shared signal, and Google, Microsoft and Anthropic made the same commitment the same day.
The signal comes from a new Internet-Draft written by Kevin Kelley of Anthropic for the Internet Engineering Task Force’s AI Preferences (aipref) working group, which has spent more than a year on a common language for AI usage preferences. Until now, staying out of AI training meant listing GPTBot, ClaudeBot, Google-Extended and every other crawler by name, and keeping that list up to date.
How to use it #
The draft adds no new commands. Instead it defines a “policy token”, __usage_ai-training, which goes in the User-Agent line of an ordinary robots.txt group. To opt a whole site out of training while leaving a press kit open, for example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Disallow: / on its own covers the whole site; a path such as /archive/ covers one section.
One detail matters for anyone who already has a robots.txt file. A catch-all User-Agent: * rule does not count as a training preference. Kelley, announcing the draft to the working group, said that was deliberate:
A robots.txt file that doesn’t use this mechanism, as is true of every one published today, expresses no preference at all.
Kevin Kelley, Anthropic, on the IETF ai-control mailing list
In other words, nothing changes for a site until its owner adds the new group.
When each company will start #
- Anthropic: Kelley said the company “has already started implementing support”, but that “it’ll take us some time to add support across our services”.
- OpenAI: it aims to implement the signal “in the coming months” and says its existingGPTBot crawler controls will keep working.
- Google: itplans to support it in 2027 . A site that opts out with the shared token, and doesn’t mention Google-Extended, will be treated as if it had blocked Google-Extended, Google’s own AI training control since 2023.
- Microsoft: it willstart honouring it on February 1, 2027 as Bing recrawls pages.
Each company’s own controls keep working alongside the shared one. A site can opt out of training across the board and then let one developer back in through that company’s own token, which Google says it will allow with Google-Extended. Meta, Apple, Amazon and xAI had not announced support by Friday evening. Meta is moving the other way: its new business terms let it crawl sites to train its AI with no opt-out.
Training only, not search or AI answers #
The token takes its meaning from the working group’s vocabulary draft, which defines AI training as “using an asset to modify the learned parameters of a generative AI model”. It does not cover search indexing, and it does not cover grounding, where an AI tool reads a page to answer a question in real time.
Microsoft is adding a separate grounding setting for Bing, Copilot and Microsoft 365, set page by page with a meta tag such as <meta name="Bingbot" content="ai-grounding:disallow">, and says neither changes eligibility for ordinary Bing results. Sites using the older NoArchive or NoCache tags to keep content out of Microsoft’s AI must switch before March 1, 2027, when Microsoft stops reading them as a preference. Google says sites that allow Google-Extended are the ones eligible to appear, with clicks, in Gemini app answers grounded in the web.
The limits #
This is a voluntary preference, not a lock. The draft says the token “is not a directive for crawling behavior”, and only companies that sign up will honour it; scrapers that ignore robots.txt will ignore this too. It is not yet a standard: the draft is an individual submission with “no formal standing” at the IETF, it expires in May 2027 unless renewed, and Kelley warned that how Anthropic applies the signal “may change” as the work goes on.
It also looks forward, not back: the preference that applies is the one in place when a page was fetched, so it does nothing about content already collected, the issue at the heart of lawsuits such as the one Canada’s biggest newspapers filed against Anthropic. Private licensing deals can override it.
Why it matters #
For the first time the biggest AI developers have agreed on one switch that any site owner can flip without tracking dozens of crawler names. It only works if others join and the companies keep their word, and it leaves untouched the AI answers that now sit between publishers and their readers.