One robots.txt Rule to Opt Out of AI Training: What Developers Need to Know (2027)

Google announced on October 30, 2026 that it will support a shared AI training control in robots.txt, built on work from the IETF's AIPREF working group, with Microsoft and OpenAI reported to honor it and Anthropic building support. The mechanism uses a policy token, __usage_ai-training, on the User-Agent line rather than a crawler name, so standard robots.txt path matching lets site owners opt out site-wide while allowing specific folders. Google said opting out via the shared token without explicitly naming Google-Extended will be treated as disallowing it, while sites that allow Google-Extended remain eligible for grounding and citations in Gemini Apps.

On October 30, 2026, Google announced plans to support a shared AI training control in robots.txt , based on work from the IETF's AIPREF working group. Coverage the same day reported that Microsoft and OpenAI will honour it too, and that Anthropic has started building support. If you build or maintain websites, here's what it means and what to do all free . User-Agent: usage ai-training Disallow: / Instead of a crawler name, the User-Agent line holds a policy token . Normal robots.txt path matching applies, so you can opt out site-wide and still allow a folder: User-Agent: usage ai-training Disallow: / Allow: /press-kit/ It's an IETF draft, so the exact token or details could still change. Google says that if you use the shared token to opt out and don't explicitly specify Google-Extended , it will treat that as if you disallowed Google-Extended. Google also says sites that allow Google-Extended are eligible to appear and receive clicks via grounding in Gemini Apps. So if you want to opt out broadly but still be cited in Gemini, you'd add an explicit allow, knowing that it also lets Google use the content for Gemini: Opt out of AI training with all participating AI providers User-Agent: usage ai-training Disallow: / Optional: still allow Google's Gemini training + citations in Gemini Apps User-Agent: Google-Extended Allow: / yoursite.com/robots.txt and see what's already there. What would you choose for your projects: block, allow, or a mix by folder?