{"slug": "machine-readable-training-preference-signal", "title": "Machine-Readable Training Preference Signal", "summary": "An Internet-Draft by Kelley published in October 2026 proposes letting service operators bind usage preferences such as \"no AI training\" to specific paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line. The draft, which expires 3 May 2027 and is discussed on the IETF AI Preferences Working Group mailing list, introduces no new directive names or grammar, leaving robots.txt path language, longest-match precedence and group structure unchanged so existing parsers ignore policy-token groups as they would any unrecognized user agent. The approach lets a site express a preference against AI training for all paths except those under /press-kit/, where it expresses a preference in favor.", "body_md": "| Internet-Draft | Training Preference Signal | October 2026 | \n| Kelley | Expires 3 May 2027 | [Page] | \n\nThis document describes how a service operator binds usage preferences\n(e.g., \"no AI training\") to paths by writing ordinary `robots.txt`\ngroups keyed by a per-purpose policy token in the `User-Agent` line.\nIt introduces no new directive names and no new grammar: the path\nlanguage, longest-match precedence, and group structure of `robots.txt`\nare not changed.[¶](#section-abstract-1)\n\nThis note is to be removed before publishing as an RFC.[¶](#section-note.1-1)\n\n        Status information for this document may be found at [https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/).[¶](#section-note.1-2)\n\n        Discussion of this document takes place on the\n        AI Preferences Working Group mailing list ([mailto:ai-control@ietf.org](mailto:ai-control@ietf.org)),\n        which is archived at [https://mailarchive.ietf.org/arch/browse/ai-control/](https://mailarchive.ietf.org/arch/browse/ai-control/).\n        Subscribe at [https://www.ietf.org/mailman/listinfo/ai-control/](https://www.ietf.org/mailman/listinfo/ai-control/).[¶](#section-note.1-3)\n\n        This Internet-Draft is submitted in full conformance with the\n        provisions of BCP 78 and BCP 79.[¶](#section-boilerplate.1-1)\n\n        Internet-Drafts are working documents of the Internet Engineering Task\n        Force (IETF). Note that other groups may also distribute working\n        documents as Internet-Drafts. The list of current Internet-Drafts is\n        at [https://datatracker.ietf.org/drafts/current/](https://datatracker.ietf.org/drafts/current/).[¶](#section-boilerplate.1-2)\n\n        Internet-Drafts are draft documents valid for a maximum of six months\n        and may be updated, replaced, or obsoleted by other documents at any\n        time. It is inappropriate to use Internet-Drafts as reference\n        material or to cite them other than as \"work in progress.\"[¶](#section-boilerplate.1-3)\n\n        This Internet-Draft will expire on 3 May 2027.[¶](#section-boilerplate.1-4)\n\n            Copyright (c) 2026 IETF Trust and the persons identified as the\n            document authors. All rights reserved.[¶](#section-boilerplate.2-1)\n\n            This document is subject to BCP 78 and the IETF Trust's Legal\n            Provisions Relating to IETF Documents\n            ([https://trustee.ietf.org/license-info](https://trustee.ietf.org/license-info)) in effect on the date of\n            publication of this document. Please review these documents\n            carefully, as they describe your rights and restrictions with\n            respect to this document. Code Components extracted from this\n            document must include Revised BSD License text as described in\n            Section 4.e of the Trust Legal Provisions and are provided without\n            warranty as described in the Revised BSD License.[¶](#section-boilerplate.2-2)\n\nA service operator binds usage preferences (e.g., \"no AI training\") to\npaths by writing ordinary `robots.txt` groups keyed by a\nper-purpose **policy token** in the `User-Agent` line.[¶](#section-1-1)\n\nThis expresses a preference *against* AI training for all paths\nexcept those under `/press-kit/`, for which it expresses a preference\n*in favor* ([Section 4.2](#checking)).[¶](#section-1-3)\n\nThis document introduces no new directive names and no new grammar; the\npath language, longest-match precedence, and group structure defined by\n[[ROBOTS](#ROBOTS)] are not changed.  As a result, the wide variety of existing\nparsers deployed across the Internet will continue to interpret\ntraffic management signals as they did previously, and will ignore\npolicy-token groups as they would any group for a user agent they do\nnot recognize.[¶](#section-1-4)\n\n**Path-specific bindings.** Preferences can be expressed per path,\nand not only site-wide ([Section 4.2](#checking)).[¶](#section-1.1-1.1.1)\n\n**Minimal distance from RFC 9309.** Semantics, and ideally\nsyntax, stay as close to [[ROBOTS](#ROBOTS)] as possible, so that authors and\nimplementers can use the mental model they already have.[¶](#section-1.1-1.2.1)\n\n**No large-scale duplication.** `robots.txt` files are\nsize-constrained ([Section 2.5](https://rfc-editor.org/rfc/rfc9309#section-2.5) of [[ROBOTS](#ROBOTS)]), and some deployed\nfiles already contain heavily duplicated groups.  Expressing\npreferences does not require restating existing rules\n([Section 4.1](#policy-token)).[¶](#section-1.1-1.3.1)\n\n**Unchanged legacy behavior.** A parser or consumer that does not\nimplement this mechanism behaves as before.  This holds for parsers\nthat are actually deployed, not only for idealized RFC 9309\nparsers.  Conversely, implementations treat files that do not use\nthis mechanism exactly as before ([Section 4.2](#checking) and\n[Section 4.3](#independence)).[¶](#section-1.1-1.4.1)\n\n**Consistent interpretation across consumers.** This mechanism does\nnot depend on the divergent parts of deployed parser behavior\n([Section 4.1](#policy-token)).[¶](#section-1.1-1.5.1)\n\nThe key words \"MUST\", \"MUST NOT\", \"REQUIRED\", \"SHALL\", \"SHALL NOT\", \"SHOULD\", \"SHOULD NOT\", \"RECOMMENDED\", \"NOT RECOMMENDED\",\n\"MAY\", and \"OPTIONAL\" in this document are to be interpreted as\ndescribed in BCP 14 [[RFC2119](#RFC2119)] [[RFC8174](#RFC8174)] when, and only when, they\nappear in all capitals, as shown here.[¶](#section-2-1)\n\nAn entity that operates an HTTP service, such as a website, and\npublishes a `robots.txt` file for it.  For preferences expressed\nthrough this mechanism, the service operator is the \"declaring\nparty\" ([Section 2](https://datatracker.ietf.org/doc/html/draft-ietf-aipref-vocab-08#section-2) of [[VOCAB](#VOCAB)]).  [[ROBOTS](#ROBOTS)] calls this party a\n\"service owner\".[¶](#section-2-2.2.1)\n\nAn entity that interprets preference signals that a service operator\nexpresses through the mechanism defined in this document.[¶](#section-2-2.4.1)\n\nIn the sense of [[ROBOTS](#ROBOTS)]; software that parses `robots.txt` files\nand matches product tokens and paths against their rules.[¶](#section-2-2.6.1)\n\nA product token ([Section 2.2.1](https://rfc-editor.org/rfc/rfc9309#section-2.2.1) of [[ROBOTS](#ROBOTS)]) whose RFC 9309 signals\na consumer uses to determine how content will be used.  A policy\ntoken need not appear in the `User-Agent` header field\n([Section 10.1.5](https://rfc-editor.org/rfc/rfc9110#section-10.1.5) of [[HTTP](#HTTP)]) of any outbound request.  This document\nintroduces a new, cross-vendor policy token ([Section 4.1](#policy-token)).[¶](#section-2-2.8.1)\n\nA policy token documented by a single consumer.  Some service\noperators already include these in their `robots.txt` files.[¶](#section-2-2.10.1)\n\nThe `__usage_ai-training` policy token corresponds to the definition\nthat the IETF aipref Working Group has arrived at\n([Section 4.1](https://datatracker.ietf.org/doc/html/draft-ietf-aipref-vocab-08#section-4.1) of [[VOCAB](#VOCAB)]).  The following definitions are quoted from\nthat document:[¶](#section-3-1)\n\nUsing an asset to modify the learned parameters of a generative AI\nmodel.[¶](#section-3-2.2.1)\n\nThe training of models that are used to perform exclusively\nnon-generative tasks is not included in this category, even if the\nmodel is capable of generative tasks.[¶](#section-3-2.2.2)\n\nAn AI model that is used to generate synthetic content in one or more\nmodalities. Alternatively, an AI model that is made available for use\nto generate synthetic content in one or more modalities. This\ndefinition does not include a model that is used, or made available\nfor use, for classification, ranking, or scoring, even where the model\ngenerates rationale for its output.[¶](#section-3-2.4.1)\n\nThis document applies those definitions as follows:[¶](#section-3-3)\n\nThe content retrieved from a path.  The preferences that apply to an\nasset are those expressed in the `robots.txt` file in effect when the\nasset was retrieved ([Section 2.4](https://rfc-editor.org/rfc/rfc9309#section-2.4) of [[ROBOTS](#ROBOTS)]; see also [[ATTACH](#ATTACH)]).[¶](#section-3-4.2.1)\n\nThis document does not define a parallel vocabulary; it tracks the\nwork of the aipref Working Group.  Implementers SHOULD consider\nupdating their definitions, and adding or removing vocabulary terms,\nas the aipref Working Group reaches consensus in additional areas.[¶](#section-3-5)\n\nThis document defines the policy token `__usage_ai-training`.[¶](#section-4.1-1)\n\nThe policy token conforms to the product-token syntax in\n[Section 2.2.1](https://rfc-editor.org/rfc/rfc9309#section-2.2.1) of [[ROBOTS](#ROBOTS)] (letters, `_`, `-`).[¶](#section-4.1-2)\n\nConsumers MUST match policy tokens exactly, apart from case, as for\nany product token.  (Note that some deployed parsers match more\nflexibly; the `__` prefix makes accidental matches unlikely.)[¶](#section-4.1-3)\n\nA single group MAY name several policy tokens (consecutive\n`User-Agent` lines; [Section 2.2](https://rfc-editor.org/rfc/rfc9309#section-2.2) of [[ROBOTS](#ROBOTS)]) to bind its rules to\nseveral purposes.  Policy tokens and other product tokens MAY be\nmixed in a group.[¶](#section-4.1-4)\n\nTo determine whether a preference has been expressed for a given path,\na consumer looks up the path and policy-token combination as specified\nin Sections [2.2.1](https://rfc-editor.org/rfc/rfc9309#section-2.2.1) and [2.2.2](https://rfc-editor.org/rfc/rfc9309#section-2.2.2) of [[ROBOTS](#ROBOTS)].[¶](#section-4.2-1)\n\nHowever, wildcard groups (i.e., `User-Agent: *`) MUST NOT be interpreted\nas matching the policy token.[¶](#section-4.2-2)\n\nIf the lookup finds that the path is disallowed, a preference against\nthe use described by the policy token has been expressed for that path.\nIf it finds that the path is allowed, a preference in favor of that use\nhas been expressed.[¶](#section-4.2-3)\n\nIf no group contains the `__usage_ai-training` policy token (i.e.,\nthere is no `User-Agent: __usage_ai-training` line in the file), then\nno preference has been expressed.  If such a file contains a\nproprietary policy token, consumers SHOULD interpret that token as\nthey would in the absence of this document, but MUST NOT treat it as\nexpressing a preference in the context of this document (see\n[Section 4.5](#precedence)).[¶](#section-4.2-4)\n\nConsumers MUST interpret the RFC 9309 signals being expressed based on\ntheir own product token(s) exactly as they would in the absence of\nthis document.  That is, `ExampleBot` still checks whether there are\nrules for `ExampleBot`; if there are, they still express the service\noperator's preference about crawling.  Nothing in this document\nchanges that signaling.[¶](#section-4.3-1)\n\nContent-usage preferences and traffic management signals can both be\nexpressed in `robots.txt`, but they are independent and orthogonal.[¶](#section-4.3-2)\n\nThis policy token is not a directive for crawling behavior.[¶](#section-4.4-1)\n\nA consumer MAY use preferences to inform crawling.  For example, if it\ncrawls only for purposes covered by the policy token, it MAY refrain\nfrom retrieving resources for which a preference against those\npurposes has been expressed.[¶](#section-4.4-2)\n\nIf a service operator wishes to limit crawling, it MAY use groups for\ncrawlers' product tokens in `robots.txt` to do so.[¶](#section-4.4-3)\n\nSpecific expressions SHOULD take precedence over general ones.[¶](#section-4.5-1)\n\nConsumers MAY document that they interpret rules that match\nproprietary policy tokens as being more specific than the\n`__usage_ai-training` policy token.[¶](#section-4.5-2)\n\nPrivate agreements MAY override preferences expressed via this\nmechanism.[¶](#section-4.5-3)\n\nThe security considerations of [Section 3](https://rfc-editor.org/rfc/rfc9309#section-3) of [[ROBOTS](#ROBOTS)] apply to this\nmechanism.[¶](#section-5-1)\n\nThis document does not introduce any new privacy considerations.[¶](#section-6-1)\n\nThis document has no IANA actions.[¶](#section-7-1)\n\nThe author is grateful to the Working Group's chairs, Mark Nottingham\n(Cloudflare) and Suresh Krishnan (Cisco), to the editors of the\nvocabulary draft, Martin Thomson (Mozilla) and Paul Keller (Open\nFuture), and to the responsible Area Director, Mike Bishop (Akamai),\nwhose incredible effort to facilitate discussion and endless patience\n(in addition to their own individual contributions) have made the\nWorking Group's progress possible.[¶](#appendix-A-1)\n\nThe author also thanks the following people for their substantial\ncontributions to shaping the Working Group's discussion and for their\npartnership.  Affiliations are listed for identification only, since\nIETF participants contribute as individuals.[¶](#appendix-A-2)\n\n| Table 1 |  | \n|---|---|\n| Name | Affiliation | \n|---|---|\n| Farzaneh Badiei | Digital Medusa | \n| Lila Bailey | Internet Archive | \n| Kevin Bankston | Center for Democracy and Technology | \n| Fabrice Canel | Microsoft | \n| Alissa Cooper | Knight-Georgetown Institute | \n| Sonia Cooper | Microsoft | \n| Glenn Deen | Comcast NBCUniversal | \n| Caleb Donaldson |  | \n| Daniel Etcovitch | Anthropic | \n| Christopher Flammang | Elsevier | \n| Max Gendler | News Corp | \n| Alex Gibson | News Corp | \n| Jordan Gimbel | Microsoft | \n| Mark Gray | OpenAI | \n| Nate Hake | Travel Lemming | \n| Gary Illyes |  | \n| Meredith Jacob | American University Washington College of Law | \n| Mirja Kühlewind | Ericsson | \n| Laurent Le Meur | EDRLab | \n| Jo Levy | Norton Law | \n| Greg Lindahl | Common Crawl Foundation | \n| Krishna Madhavan | Microsoft | \n| Sarah McKenna | Sequentum | \n| Angela Mills Wade | European Publishers Council | \n| John Mueller |  | \n| Chris Needham | BBC | \n| Elaine Newton | Amazon Web Services | \n| Victoria Noble | Electronic Frontier Foundation | \n| Sebastian Posth | Liccium | \n| Brendan Quinn | IPTC | \n| Courtney Radsch | Open Markets Institute | \n| Felix Reda | GitHub | \n| Eric Rescorla | Independent | \n| Matt Rogerson | Financial Times | \n| Leah Romm | Cloudflare | \n| Leonard Rosenthol | Adobe | \n| Malikat Rufai | Meta | \n| Achim Schlosser | Bertelsmann | \n| Rony Shalit | Bright Data | \n| Bradley Silver | Advance | \n| Erin Simon |  | \n| Esther Tetruashvily | OpenAI | \n| Michael Tremante | Cloudflare | \n| Thom Vaughan | Common Crawl Foundation | \n| James Whymark | Meta | \n| Timid Robot Zehta | Creative Commons | \n\nMany more people shaped the Working Group's discussion than appear\nabove, and this list has almost certainly left out someone who belongs\non it.  Any such omission is inadvertent, and the author apologizes in\nadvance.[¶](#appendix-A-4)", "url": "https://wpnews.pro/news/machine-readable-training-preference-signal", "canonical_source": "https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html", "published_at": "2026-11-01 18:04:11+00:00", "updated_at": "2026-11-01 18:21:18.476489+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "ai-policy", "artificial-intelligence"], "entities": ["IETF", "AI Preferences Working Group", "Kelley", "Internet-Draft"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/machine-readable-training-preference-signal", "markdown": "https://wpnews.pro/news/machine-readable-training-preference-signal.md", "text": "https://wpnews.pro/news/machine-readable-training-preference-signal.txt", "jsonld": "https://wpnews.pro/news/machine-readable-training-preference-signal.jsonld"}}