On October 30, 2026, Google announced plans to support a shared AI training control in robots.txt, based on work from the IETF's AIPREF working group. Coverage the same day reported that Microsoft and OpenAI will honour it too, and that Anthropic has started building support.
If you build or maintain websites, here's what it means and what to do (all free).
User-Agent: __usage_ai-training
Disallow: /
Instead of a crawler name, the User-Agent line holds a policy token. Normal robots.txt path matching applies, so you can opt out site-wide and still allow a folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
It's an IETF draft, so the exact token or details could still change.
Google says that if you use the shared token to opt out and don't explicitly specify Google-Extended, it will treat that as if you disallowed Google-Extended. Google also says sites that allow Google-Extended are eligible to appear and receive clicks via grounding in Gemini Apps.
So if you want to opt out broadly but still be cited in Gemini, you'd add an explicit allow, knowing that it also lets Google use the content for Gemini:
User-Agent: __usage_ai-training
Disallow: /
User-Agent: Google-Extended
Allow: /
yoursite.com/robots.txt and see what's already there.
What would you choose for your projects: block, allow, or a mix by folder?