{"slug": "openai-google-microsoft-and-anthropic-will-obey-one-robots-txt-line-that-opts-of", "title": "OpenAI, Google, Microsoft and Anthropic will obey one robots.txt line that opts your site out of AI training", "summary": "OpenAI, Google, Microsoft and Anthropic all committed on Friday to honouring a single new robots.txt policy token, `__usage_ai-training`, that lets site owners opt out of AI training without blocking each company's crawler individually. The signal is defined in an Internet-Draft by Kevin Kelley of Anthropic for the IETF's AI Preferences (aipref) working group; Anthropic said it \"has already started implementing support,\" OpenAI targets implementation \"in the coming months,\" Google plans support in 2027, and Microsoft will start honouring it on February 1, 2027. Meta, Apple, Amazon and xAI had not announced support by Friday evening, and Meta's new business terms let it crawl sites to train its AI with no opt-out.", "body_md": "OpenAI, Google, Microsoft and Anthropic have all said they will honour the shared AI training signal in robots.txt. Image: OpenAI and Microsoft logos via LobeHub Icons (MIT), Google and Anthropic logos via Simple Icons (CC0); trademarks of their owners\n\nOpenAI, Google, Microsoft and Anthropic have agreed to honour a single new line in a website’s robots.txt file that says “don’t train your AI on this”, so site owners will no longer have to block each company’s crawler one by one. OpenAI [announced](https://openai.com/index/helping-websites-express-ai-training-preferences/) on Friday that it will respect the shared signal, and Google, Microsoft and Anthropic made the same commitment the same day.\n\nThe signal comes from a new [Internet-Draft](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) written by Kevin Kelley of Anthropic for the Internet Engineering Task Force’s AI Preferences (aipref) working group, which has spent more than a year on a common language for AI usage preferences. Until now, staying out of AI training meant [listing GPTBot, ClaudeBot, Google-Extended and every other crawler](https://madrobot.blog/2026/10/02/how-to-block-ai-agents-bots-website-robots-txt-gptbot-claudebot/) by name, and keeping that list up to date.\n\n## How to use it\n\nThe draft adds no new commands. Instead it defines a “policy token”, `__usage_ai-training`, which goes in the User-Agent line of an ordinary robots.txt group. To opt a whole site out of training while leaving a press kit open, for example:\n\n```\nUser-Agent: __usage_ai-training\nDisallow: /\nAllow: /press-kit/\n```\n\n`Disallow: /` on its own covers the whole site; a path such as `/archive/` covers one section.\n\nOne detail matters for anyone who already has a robots.txt file. A catch-all `User-Agent: *` rule does not count as a training preference. Kelley, announcing the draft to the working group, said that was deliberate:\n\nA robots.txt file that doesn’t use this mechanism, as is true of every one published today, expresses no preference at all.\n\nKevin Kelley, Anthropic, on the IETF ai-control mailing list\n\nIn other words, nothing changes for a site until its owner adds the new group.\n\n## When each company will start\n\n- **Anthropic:** Kelley said the company “has already started implementing support”, but that “it’ll take us some time to add support across our services”.\n- **OpenAI:** it aims to implement the signal “in the coming months” and says its existing[GPTBot crawler controls](https://developers.openai.com/api/docs/bots) will keep working.\n- **Google:** it[plans to support it in 2027](https://developers.google.com/search/blog/2026/10/ai-training-control) . A site that opts out with the shared token, and doesn’t mention Google-Extended, will be treated as if it had blocked Google-Extended, Google’s own AI training control since 2023.\n- **Microsoft:** it will[start honouring it on February 1, 2027](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) as Bing recrawls pages.\n\nEach company’s own controls keep working alongside the shared one. A site can opt out of training across the board and then let one developer back in through that company’s own token, which Google says it will allow with Google-Extended. Meta, Apple, Amazon and xAI had not announced support by Friday evening. Meta is moving the other way: its new business terms [let it crawl sites to train its AI with no opt-out](https://madrobot.blog/2026/10/16/meta-commercial-terms-crawl-websites-train-ai-publishers-no-opt-out/).\n\n## Training only, not search or AI answers\n\nThe token takes its meaning from the working group’s [vocabulary draft](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/), which defines AI training as “using an asset to modify the learned parameters of a generative AI model”. It does not cover search indexing, and it does not cover grounding, where an AI tool reads a page to answer a question in real time.\n\nMicrosoft is adding a separate grounding setting for Bing, Copilot and Microsoft 365, set page by page with a meta tag such as `<meta name=\"Bingbot\" content=\"ai-grounding:disallow\">`, and says neither changes eligibility for ordinary Bing results. Sites using the older NoArchive or NoCache tags to keep content out of Microsoft’s AI must switch before March 1, 2027, when Microsoft stops reading them as a preference. Google says sites that allow Google-Extended are the ones eligible to appear, with clicks, in Gemini app answers grounded in the web.\n\n## The limits\n\nThis is a voluntary preference, not a lock. The draft says the token “is not a directive for crawling behavior”, and only companies that sign up will honour it; scrapers that ignore robots.txt will ignore this too. It is not yet a standard: the draft is an individual submission with “no formal standing” at the IETF, it expires in May 2027 unless renewed, and Kelley warned that how Anthropic applies the signal “may change” as the work goes on.\n\nIt also looks forward, not back: the preference that applies is the one in place when a page was fetched, so it does nothing about content already collected, the issue at the heart of lawsuits such as the one [Canada’s biggest newspapers filed against Anthropic](https://madrobot.blog/2026/10/19/canadian-news-publishers-sue-anthropic-ontario-globe-mail-toronto-star-postmedia/). Private licensing deals can override it.\n\n## Why it matters\n\nFor the first time the biggest AI developers have agreed on one switch that any site owner can flip without tracking dozens of crawler names. It only works if others join and the companies keep their word, and it leaves untouched the AI answers that now sit between publishers and their readers.", "url": "https://wpnews.pro/news/openai-google-microsoft-and-anthropic-will-obey-one-robots-txt-line-that-opts-of", "canonical_source": "https://madrobot.blog/2026/10/30/robots-txt-ai-training-opt-out-signal/", "published_at": "2026-10-30 21:12:00+00:00", "updated_at": "2026-10-30 21:20:31.012338+00:00", "lang": "en", "topics": ["ai-policy", "ai-crawlers", "agent-protocols", "artificial-intelligence"], "entities": ["OpenAI", "Google", "Microsoft", "Anthropic", "Kevin Kelley", "Internet Engineering Task Force", "GPTBot", "Google-Extended"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/openai-google-microsoft-and-anthropic-will-obey-one-robots-txt-line-that-opts-of", "markdown": "https://wpnews.pro/news/openai-google-microsoft-and-anthropic-will-obey-one-robots-txt-line-that-opts-of.md", "text": "https://wpnews.pro/news/openai-google-microsoft-and-anthropic-will-obey-one-robots-txt-line-that-opts-of.txt", "jsonld": "https://wpnews.pro/news/openai-google-microsoft-and-anthropic-will-obey-one-robots-txt-line-that-opts-of.jsonld"}}