# OpenAI, Google, Microsoft and Anthropic will obey one robots.txt line that opts your site out of AI training

> Source: <https://madrobot.blog/2026/10/30/robots-txt-ai-training-opt-out-signal/>
> Published: 2026-10-30 21:12:00+00:00

OpenAI, Google, Microsoft and Anthropic have all said they will honour the shared AI training signal in robots.txt. Image: OpenAI and Microsoft logos via LobeHub Icons (MIT), Google and Anthropic logos via Simple Icons (CC0); trademarks of their owners

OpenAI, Google, Microsoft and Anthropic have agreed to honour a single new line in a website’s robots.txt file that says “don’t train your AI on this”, so site owners will no longer have to block each company’s crawler one by one. OpenAI [announced](https://openai.com/index/helping-websites-express-ai-training-preferences/) on Friday that it will respect the shared signal, and Google, Microsoft and Anthropic made the same commitment the same day.

The signal comes from a new [Internet-Draft](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) written by Kevin Kelley of Anthropic for the Internet Engineering Task Force’s AI Preferences (aipref) working group, which has spent more than a year on a common language for AI usage preferences. Until now, staying out of AI training meant [listing GPTBot, ClaudeBot, Google-Extended and every other crawler](https://madrobot.blog/2026/10/02/how-to-block-ai-agents-bots-website-robots-txt-gptbot-claudebot/) by name, and keeping that list up to date.

## How to use it

The draft adds no new commands. Instead it defines a “policy token”, `__usage_ai-training`, which goes in the User-Agent line of an ordinary robots.txt group. To opt a whole site out of training while leaving a press kit open, for example:

```
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```

`Disallow: /` on its own covers the whole site; a path such as `/archive/` covers one section.

One detail matters for anyone who already has a robots.txt file. A catch-all `User-Agent: *` rule does not count as a training preference. Kelley, announcing the draft to the working group, said that was deliberate:

A robots.txt file that doesn’t use this mechanism, as is true of every one published today, expresses no preference at all.

Kevin Kelley, Anthropic, on the IETF ai-control mailing list

In other words, nothing changes for a site until its owner adds the new group.

## When each company will start

- **Anthropic:** Kelley said the company “has already started implementing support”, but that “it’ll take us some time to add support across our services”.
- **OpenAI:** it aims to implement the signal “in the coming months” and says its existing[GPTBot crawler controls](https://developers.openai.com/api/docs/bots) will keep working.
- **Google:** it[plans to support it in 2027](https://developers.google.com/search/blog/2026/10/ai-training-control) . A site that opts out with the shared token, and doesn’t mention Google-Extended, will be treated as if it had blocked Google-Extended, Google’s own AI training control since 2023.
- **Microsoft:** it will[start honouring it on February 1, 2027](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) as Bing recrawls pages.

Each company’s own controls keep working alongside the shared one. A site can opt out of training across the board and then let one developer back in through that company’s own token, which Google says it will allow with Google-Extended. Meta, Apple, Amazon and xAI had not announced support by Friday evening. Meta is moving the other way: its new business terms [let it crawl sites to train its AI with no opt-out](https://madrobot.blog/2026/10/16/meta-commercial-terms-crawl-websites-train-ai-publishers-no-opt-out/).

## Training only, not search or AI answers

The token takes its meaning from the working group’s [vocabulary draft](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/), which defines AI training as “using an asset to modify the learned parameters of a generative AI model”. It does not cover search indexing, and it does not cover grounding, where an AI tool reads a page to answer a question in real time.

Microsoft is adding a separate grounding setting for Bing, Copilot and Microsoft 365, set page by page with a meta tag such as `<meta name="Bingbot" content="ai-grounding:disallow">`, and says neither changes eligibility for ordinary Bing results. Sites using the older NoArchive or NoCache tags to keep content out of Microsoft’s AI must switch before March 1, 2027, when Microsoft stops reading them as a preference. Google says sites that allow Google-Extended are the ones eligible to appear, with clicks, in Gemini app answers grounded in the web.

## The limits

This is a voluntary preference, not a lock. The draft says the token “is not a directive for crawling behavior”, and only companies that sign up will honour it; scrapers that ignore robots.txt will ignore this too. It is not yet a standard: the draft is an individual submission with “no formal standing” at the IETF, it expires in May 2027 unless renewed, and Kelley warned that how Anthropic applies the signal “may change” as the work goes on.

It also looks forward, not back: the preference that applies is the one in place when a page was fetched, so it does nothing about content already collected, the issue at the heart of lawsuits such as the one [Canada’s biggest newspapers filed against Anthropic](https://madrobot.blog/2026/10/19/canadian-news-publishers-sue-anthropic-ontario-globe-mail-toronto-star-postmedia/). Private licensing deals can override it.

## Why it matters

For the first time the biggest AI developers have agreed on one switch that any site owner can flip without tracking dozens of crawler names. It only works if others join and the companies keep their word, and it leaves untouched the AI answers that now sit between publishers and their readers.