{"slug": "content-signals-in-robots-txt", "title": "Content Signals in robots.txt", "summary": "Content Signals, a proposed extension to robots.txt backed by the IETF AI Preferences working group and the IAB Tech Lab, adds directives to declare whether AI crawlers may search, ingest, or train on content. The IETF attachment draft expired on 1 May 2026, leaving the syntax resting on the IAB Tech Lab specification and validator convention, though validators like isitagentready.com already check for it. Sites can implement per-group directives such as 'Content-Signal: search=yes, ai-input=yes, ai-train=no' to control AI access.", "body_md": "# Content Signals in robots.txt\n\nAdd Content-Signal directives to robots.txt to declare whether AI crawlers may search, ingest, or train on your content. An emerging IETF AI Preferences / IAB Tech Lab proposal that some validators already check.\n\n## What it is\n\nContent Signals is a proposed extension to `robots.txt`\n\nthat adds new directives expressing how a site wants its content treated downstream — specifically by AI systems. The directives live in normal `robots.txt`\n\ngroups and declare boolean preferences such as “you may use this for search” or “you may not train on this”.\n\n```\nUser-agent: *\nAllow: /\nContent-Signal: search=yes, ai-input=yes, ai-train=no\n```\n\nThe three values that are emerging as canonical:\n\n— whether the content may be indexed by search engines.`search`\n\n— whether the content may be fetched as live input to an AI system (retrieval-augmented generation, summarisation at query time).`ai-input`\n\n— whether the content may be included in a training corpus.`ai-train`\n\nEach takes `yes`\n\nor `no`\n\n. Multiple values are comma-separated on a single `Content-Signal:`\n\nline.\n\n## Status of the proposal\n\nThis is **not yet a settled standard**, and the two halves of it are in very different health. The work is split across two efforts:\n\n- The\n**IETF AI Preferences working group (** is working on a vocabulary and protocol-level definition.[aipref](https://datatracker.ietf.org/wg/aipref/documents/)) - The\n**IAB Tech Lab Content Signals project** has published a parallel specification aimed at industry adoption.\n\nInside the IETF work, separate the *vocabulary* from the *attachment mechanism*. The vocabulary — what `search`\n\n, `ai-input`\n\nand `ai-train`\n\nmean — is alive and was being edited for publication as recently as April 2026 ([draft-ietf-aipref-vocab](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/)). The document that would define how you actually bind those preferences to content over HTTP, [draft-ietf-aipref-attach](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/), **expired on 1 May 2026** and has not been reposted; its text has seen no edits in the working group’s repository since September 2025. So the part of Content Signals a site can act on today — a line in `robots.txt`\n\n— currently rests on the IAB Tech Lab specification and on validator convention, not on a live IETF draft.\n\nThat is not a reason to avoid it, but it is a reason to expect the syntax to move. Treat Content Signals as recommended-to-experiment-with, not as a finalised standard. The directive will be ignored by every crawler that does not yet parse it — which today is most of them.\n\n## Why it matters\n\n**A declarative opt-in / opt-out at the right layer.** Existing`User-agent: GPTBot / Disallow:`\n\ndirectives are coarse — they block a specific bot from fetching anything. Content Signals separates*what*the bot may do from*who*the bot is.**Validators already check for it.**[isitagentready.com](https://isitagentready.com/)explicitly looks for`Content-Signal:`\n\nlines in`robots.txt`\n\n. Sites that want a clean agent-readiness scorecard should add them.**Some major model providers have signalled future support.** Cloudflare, Google, and others have published positions on the IETF drafts.\n\n## How to implement\n\n**Per-group, in robots.txt.** Place the\n\n`Content-Signal:`\n\nline inside the same group as `User-agent:`\n\nand `Allow:`\n\n/ `Disallow:`\n\n.\n\n```\nUser-agent: *\nAllow: /\nContent-Signal: search=yes, ai-input=yes, ai-train=yes\n```\n\nThe example above says: “any crawler may use this content for search, for AI input, and for AI training.” That is the right declaration for a public spec that wants to be readable.\n\n**Different signals per crawler if your policy varies.** Use a targeted group:\n\n```\nUser-agent: GPTBot\nAllow: /\nContent-Signal: search=yes, ai-input=yes, ai-train=no\n\nUser-agent: *\nAllow: /\nContent-Signal: search=yes, ai-input=yes, ai-train=yes\n```\n\n**Pair with crawler-specific blocks where the answer is “no”.** Content Signals is a hint; many crawlers still only obey `Disallow:`\n\n. A `Content-Signal: ai-train=no`\n\npaired with `User-agent: GPTBot \\n Disallow: /`\n\nis stronger than either alone.\n\n**Don’t treat it as legal force.** It is a declaration. Compliance is voluntary, and the legal status of “you used my content for training despite my Content-Signal” is still developing.\n\n## Common mistakes\n\n- Spelling:\n`Content-Signal:`\n\n(singular). Not`Content-Signals:`\n\n. - Putting the directive outside a group (no preceding\n`User-agent:`\n\nline). Some parsers will silently ignore it. - Conflicting with\n`Disallow:`\n\n. If you`Disallow: /`\n\nfor a bot, that bot was never going to fetch the page to read your`Content-Signal:`\n\n. They contradict; pick one. - Inventing values beyond\n`yes`\n\n/`no`\n\n. The vocabulary is small on purpose. - Treating it as a substitute for\n`Disallow:`\n\n. It is complementary.\n\n## Verification\n\n`curl -s https://example.com/robots.txt | grep -i content-signal`\n\nlists every directive.[Is It Agent Ready?](https://isitagentready.com/)flips`botAccessControl.contentSignals`\n\nto`pass`\n\n.- Track the\n[IETF aipref WG](https://datatracker.ietf.org/wg/aipref/documents/)for the final published vocabulary — names may yet shift before the RFC is published.\n\n## Related topics\n\n## Sources & further reading\n\n[IETF AI Preferences WG (aipref) — drafts](https://datatracker.ietf.org/wg/aipref/documents/)— IETF[IAB Tech Lab — Content Monetization Protocols (CoMP) for AI](https://iabtechlab.com/working-groups/content-monetization-protocols-comp-for-ai-working-group/)— IAB Tech Lab[Is It Agent Ready? — Content Signals check](https://isitagentready.com/)— Is It Agent Ready?[RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)— IETF", "url": "https://wpnews.pro/news/content-signals-in-robots-txt", "canonical_source": "https://specification.website/spec/agent-readiness/content-signals/", "published_at": "2026-07-28 00:00:00+00:00", "updated_at": "2026-07-31 09:58:36.838483+00:00", "lang": "en", "topics": ["ai-policy", "ai-tools"], "entities": ["IETF", "IAB Tech Lab", "Cloudflare", "Google", "isitagentready.com", "GPTBot"], "alternates": {"html": "https://wpnews.pro/news/content-signals-in-robots-txt", "markdown": "https://wpnews.pro/news/content-signals-in-robots-txt.md", "text": "https://wpnews.pro/news/content-signals-in-robots-txt.txt", "jsonld": "https://wpnews.pro/news/content-signals-in-robots-txt.jsonld"}}