One robots.txt Line Blocks All Four AI Labs From Training on Your Site

Anthropic's Kevin Kelley proposed a shared robots.txt token, `__usage_ai-training`, as an IETF Internet-Draft on October 30, 2026, giving sites one entry that tells Google, OpenAI, Microsoft and Anthropic to exclude their content from AI training. Only Anthropic honors the token today; OpenAI says it is coming in "coming months," Microsoft committed to February 1, 2027, and Google to early 2027, so sites should keep existing GPTBot, ClaudeBot and Google-Extended blocks in place. The token covers training-data collection only — it does not stop crawling, real-time retrieval or citations, or remove content already in datasets, and Meta, Apple and Amazon made no implementation commitments.

Google, OpenAI, Microsoft, and Anthropic just agreed on a single robots.txt token that tells all of them to stop training AI models on your content. Two lines. No separate entry for every crawler, no annual maintenance ritual as new bots arrive. Add this to your robots.txt and you’re done — at least for the labs that have actually shipped it. The Token and How to Use It The token is usage ai-training , proposed by Anthropic’s Kevin Kelley as an IETF Internet-Draft https://fireup.pro/news/ai-training-opt-out-robots-txt on October 30, 2026. The syntax is standard robots.txt — your existing parser handles it without changes: User-Agent: usage ai-training Disallow: / If you want to carve out specific directories — say, a press kit or public documentation — path-matching works exactly as you’d expect: User-Agent: usage ai-training Disallow: / Allow: /press-kit/ The double underscore in the token name is intentional: it prevents accidental matches by crawlers with flexible parsers. One important gotcha — a wildcard User-Agent: entry does not match this token. You have to add it explicitly. Who Honors It Now And Who Doesn’t Yet Here’s where the announcement gets less exciting. Only Anthropic is actually honoring the token today. The others have made commitments: - Anthropic — Live now - OpenAI — “Coming months” no date - Microsoft — February 1, 2027 - Google — Early 2027 That’s a good announcement and a disappointing timeline. Anthropic deserves credit for shipping instead of scheduling a press release for work it’ll do next year. The others are committing now to something that costs them nothing until 2027. Useful — but watch the gap between announcements and implementation. Until all four are live, don’t remove your legacy per-crawler entries. Keep GPTBot , ClaudeBot , and Google-Extended blocks in place alongside the new token. The shared signal complements existing rules; it doesn’t replace them until everyone ships. What This Does Not Block The usage ai-training token covers training data collection only. It does not block: - Crawling itself — AI crawlers will still visit your pages; they just can’t use the content for training - Real-time retrieval — your articles can still be cited or summarized in ChatGPT or Gemini answers, because inference is a separate pipeline - Existing training data — content already in datasets stays there; there’s no retroactive removal - Non-signatories — Meta, Apple, and Amazon participated in the working group discussions but made no implementation commitments That last point matters: you’re blocking four labs, not the entire AI industry. A new entrant that doesn’t adopt the standard can still scrape freely. Robots.txt has always operated on trust, and this token inherits that same enforcement model — voluntary, with no technical penalties for ignoring it. The Google-Extended Nuance If you use Google Search, pay attention to this. Google maps usage ai-training to its existing Google-Extended control, which covers both Gemini model training and Gemini Apps grounding. So a site-wide disallow under the shared token also blocks Google from using your content for real-time Gemini citations. If you want to block Gemini training but keep Gemini citation eligibility, add an explicit Google-Extended allow group separately. The specific rule takes precedence https://thenextweb.com/news/ai-training-opt-out-robots-txt-google-openai-microsoft-anthropic over the general one per robots.txt spec. Why This Matters Managing AI training opt-outs has been a mess. Add GPTBot, then ClaudeBot, then Google-Extended, then Applebot-Extended, then whatever next year’s new entrant names its crawler. Miss one and you’ve implicitly consented. The IETF’s AIPREF working group https://dev.to/shivaysinghrajput/one-robotstxt-line-that-speaks-to-every-ai-lab-implementing-the-shared-training-opt-out-4l7p was formed specifically to solve this fragmentation — 46 contributors including News Corp, the Financial Times, the BBC, EFF, and CDT spent time on this draft. The result isn’t a formal IETF standard yet — the Internet-Draft expires May 3, 2027, and has no formal working-group endorsement. But four major labs publicly committing to honor it is as close to a de facto standard as this space has seen. Add the token now. Keep your legacy rules. Revisit in early 2027 when the remaining three labs are supposed to be live.