# OpenAI AI Training Preferences in robots.txt: A Safe Rollout Guide for Web Teams > Source: > Published: 2026-11-02 16:01:04+00:00 A shared AI-training signal is arriving. Treat it as a versioned content-use policy — not a magic bot block — and you can keep your site’s intent clear without breaking another path by accident. One website can have more than one AI-use decision. The useful first step is to name each one. A two-line change in robots.txt can feel wonderfully simple: publish a preference, solve an AI-policy question, move on. That is exactly why it deserves more care. On October 30, OpenAI said it plans to honor a shared robots.txt signal that lets site operators state a preference about the use of content for foundational generative-AI training. Google announced support for the same direction. The proposal is based on current IETF work and uses a policy token named __usage_ai-training. The important word is *preference*. The signal is about a particular use of content: changing the learned parameters of a generative model. It is not, by itself, a general instruction to crawl or not crawl. It does not settle search inclusion, user-triggered visits, CDN bot rules, historic collection, contract terms, or every future AI use case. That distinction is not a technical footnote. It is the difference between a policy that says what your organization meant and one that accidentally changes discovery, support, analytics, or the next person’s understanding of the system. *Do not start with a line of syntax. Start with a sentence your team can defend: “We do or do not permit this part of the site to be used for generative-AI training.”* This guide gives web developers, platform teams, and technical SEO leads a safe rollout path. It does not offer legal advice. Its aim is narrower and more practical: make the live policy, delivery path, and evidence match the decision. The IETF Internet-Draft describes a way to attach a content-use preference to paths using ordinary robots.txt group syntax. The shared token is __usage_ai-training. A disallow rule expresses a preference against AI training for a matching path; an allow rule expresses a preference in favor. That draft is work in progress, not a finished RFC. OpenAI says it will share implementation details as its rollout develops. Google says it intends to support the signal in early 2027. In other words: your policy design can begin now, but production behavior should be verified against each provider’s current documentation and observed rollout status before anyone claims the change is active everywhere. What has *not* changed is just as useful: This is good news for system design. You do not have to make one overloaded “AI yes/no” switch do every job. You can name the jobs and use the control that fits each one. Most mistakes happen when an organization jumps from a policy meeting to a production file. A better pattern has four boundaries: intent, policy, delivery, and evidence. Each boundary asks a simple question, and each one catches a different kind of failure. Write the decision in plain language before opening a repository. “No AI” is too vague to implement well. Ask the content owner, legal or policy lead, product owner, and platform owner to decide separately: Those answers might all be “yes,” all be “no,” or form a mix. A documentation site may decline training for most material while allowing a small media kit. A public support center may allow search discovery but choose a separate training preference. A SaaS dashboard should normally rely on authentication and application controls, not a public crawler policy, to protect customer data. **Practical rule:** give every answer an owner. “Marketing owns public content intent; platform owns delivery; security owns bot enforcement; SEO owns search-impact review” is far safer than “engineering will update robots.” Once intent is clear, use the narrowest rule that says it. The current draft’s whole-site opt-out pattern is straightforward: ``` User-Agent: __usage_ai-trainingDisallow: / ``` For a site that wants a broad training preference against use but wishes to carve out a public asset path, the draft provides this shape: ``` User-Agent: __usage_ai-trainingDisallow: /Allow: /press-kit/ ``` Do not copy these examples into production and then delete existing named-agent groups. The shared token is designed to be orthogonal to those groups. Keep a separate block for each existing policy that still matters. If your team already has a rule for GPTBot, preserve or deliberately revise it based on its own meaning. If you use Google-Extended, review it too: Google says the new shared opt-out will be interpreted as a Google-Extended disallow when Google-Extended is not explicitly specified, but your explicit configuration should still communicate your intent clearly. Also avoid an easy trap from ordinary crawler thinking. The draft says a wildcard group such as User-Agent: * does *not* express this new policy preference. The shared policy token needs its own group. A global block may affect conventional crawler access, but it is not a substitute for documenting a training preference. The policy file is only one stage. Keep training preference, discovery behavior, and release verification on separate tracks. A correct commit can still produce a wrong site. robots.txt is often assembled through a framework route, CMS plugin, static build, CDN edge rule, hostname redirect, or WAF setting. Each layer is an opportunity for drift. Start by finding the source of truth. Is the file committed in the repository? Generated from a CMS checkbox? Written by a framework handler? Overridden by a platform? Then map the public endpoints that matter: apex domain, www, preview host, alternate locale host, and any content subdomain. A policy that exists only on one hostname may not represent what you think it represents. Next, fetch the artifact like a client would. Do not rely only on a local file. ``` curl -i https://example.com/robots.txtcurl -sS https://example.com/robots.txt | sed -n '1,160p' # Follow the public redirect path too.curl -IL https://www.example.com/robots.txt ``` Check the status code, redirect destination, content type, cache headers, and actual body. In CI, a lightweight assertion can catch accidental removal during a theme, CDN, or application deployment: ``` curl -fsS https://example.com/robots.txt \ | grep -Fqx 'User-Agent: __usage_ai-training' curl -fsS https://example.com/robots.txt \ | grep -Fqx 'Disallow: /' ``` That example is intentionally simple. In a real check, parse groups rather than grepping the whole file, because another group may contain a matching line with a different meaning. The point is to turn a sensitive policy into a release assertion, not a memory test. A policy needs a receipt. Store a short change record beside the configuration or in the ticket that introduced it. Include the decision owner, the exact paths covered, named-agent rules reviewed, delivery endpoints checked, review date, and a link to the provider documentation used at that time. Then track evidence that answers the questions you can actually answer. Web logs can show requests, bot verification status, paths, response codes, rate patterns, and whether a CDN blocked traffic. They cannot prove that every third party complied with a future-use preference. A snapshot of a provider’s documentation can show stated support, not the contents of a training corpus. That may sound modest. It is also honest and operationally useful. Your goal is not to promise perfect visibility into another organization’s systems. Your goal is to keep your own intent, controls, and escalation path clear. Teams commonly use “AI crawler” as if it were one thing. It is a category, not a control plane. Separate these cases in your runbook. This is the new shared signal’s job. It speaks to use of retrieved content for training generative models. It is a content-use policy. It is not a rate-limit setting, search command, or authentication boundary. Discovery can have product-specific rules. OpenAI’s crawler documentation distinguishes OAI-SearchBot from GPTBot. If your publishing strategy values appearing in search-assisted answers, do not make a training-policy change without checking that the separate discovery policy still says what you intend. Conversely, allowing a search bot does not equal opting in to training. If the requirement is “this client must not reach this endpoint,” use enforcement controls: authentication, authorization, network filtering, a WAF, bot verification, rate limits, or a CDN rule. Validate claimed bots against providers’ published verification guidance where available. Blocking on a user-agent string alone is weak because user agents can be spoofed. Keeping these controls separate reduces two bad outcomes: a public policy being treated as a security mechanism, and a security rule being mistaken for a content-use preference. Use this as a pull-request checklist or a short change ticket. It is deliberately small enough to survive real releases. For larger sites, add a policy test fixture. Give it a sample robots.txt with a broad disallow, a specific allow exception, legacy crawler groups, and a wildcard group. Assert that your parser and release tooling preserve each group. This catches an especially common failure: a “cleanup” change that flattens distinct policies into one block. Release confidence comes from checking the live artifact and its delivery path, not from trusting a local file. The shared signal is new. Provider support, documented semantics, and the IETF draft can change. That makes this a configuration with a review cycle, not an eternal snippet to paste into a theme. Set a calendar reminder tied to the rollout status of the providers you care about. Recheck the official documentation when OpenAI publishes implementation details, when Google begins support, when your CDN changes bot settings, and when an agency or plugin proposes a “helpful” robots update. Treat a platform migration, new domain, CMS redesign, or SEO audit as a trigger for the same review. A simple policy manifest can make that durable: ``` policy: ai-training-preferenceowner: web-platform@example.comscope: public marketing and docs hostsintent: disallow generative-AI training except /press-kit/separate_reviews: - automated search discovery - user-initiated fetches - bot enforcement and rate limitsverified_live: 2026-11-01next_review: provider-rollout-or-next-platform-change ``` Keep that manifest as an internal record; it is not a proposed web standard. Its value is human: when a future teammate asks why a strange token exists, the answer is close to the code and connected to a decision. The shared training-preference signal is useful precisely because it narrows a messy problem. You no longer have to maintain only a long list of vendor-specific names to express a broad training choice. But a broad choice still needs careful boundaries. Decide what use of content you mean. Express the smallest correct policy. Preserve separate crawler and discovery controls. Verify the live file through the real delivery stack. Record evidence without overstating what it can prove. That is how a tiny robots.txt change becomes a trustworthy web-platform policy rather than a mysterious line everyone is afraid to touch. It is the shared policy token described by an IETF Internet-Draft for expressing a preference about use of website content in generative-AI training. It appears in a dedicated robots.txt group. The draft is work in progress, so check current provider documentation before relying on implementation behavior. No. The draft explicitly separates content-use preferences from crawler traffic management. Use named crawler groups if you want to express a crawler preference, and use authentication, a WAF, rate limits, or other enforcement controls when you need to restrict access. These are separate decisions. OpenAI documents GPTBot for potential foundational-model training use and OAI-SearchBot for automated search crawling. Review each relevant rule independently so your configuration matches your intended discovery policy. Not automatically. The shared signal is designed to coexist with existing controls, and providers can document their own precedence rules. Preserve existing groups until your team deliberately decides how each one should behave and verifies current provider guidance. No. A training preference is not a universal control over search, grounding, user-initiated fetches, prior collection, third-party datasets, or every AI product. Avoid guarantees. Document the specific use you are expressing a preference about and maintain separate controls for other goals. Fetch the public /robots.txt on every production hostname, inspect the redirect chain and final body, and save that evidence with the change record. Also check the CMS, CDN, and WAF layers that can rewrite or override the file. Revisit the test when a provider rollout or platform change occurs. [OpenAI AI Training Preferences in robots.txt: A Safe Rollout Guide for Web Teams](https://pub.towardsai.net/openai-ai-training-preferences-in-robots-txt-a-safe-rollout-guide-for-web-teams-13603478ee19) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.