# Content Signals in robots.txt

> Source: <https://specification.website/spec/agent-readiness/content-signals/>
> Published: 2026-07-28 00:00:00+00:00

# Content Signals in robots.txt

Add Content-Signal directives to robots.txt to declare whether AI crawlers may search, ingest, or train on your content. An emerging IETF AI Preferences / IAB Tech Lab proposal that some validators already check.

## What it is

Content Signals is a proposed extension to `robots.txt`

that adds new directives expressing how a site wants its content treated downstream — specifically by AI systems. The directives live in normal `robots.txt`

groups and declare boolean preferences such as “you may use this for search” or “you may not train on this”.

```
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
```

The three values that are emerging as canonical:

— whether the content may be indexed by search engines.`search`

— whether the content may be fetched as live input to an AI system (retrieval-augmented generation, summarisation at query time).`ai-input`

— whether the content may be included in a training corpus.`ai-train`

Each takes `yes`

or `no`

. Multiple values are comma-separated on a single `Content-Signal:`

line.

## Status of the proposal

This is **not yet a settled standard**, and the two halves of it are in very different health. The work is split across two efforts:

- The
**IETF AI Preferences working group (** is working on a vocabulary and protocol-level definition.[aipref](https://datatracker.ietf.org/wg/aipref/documents/)) - The
**IAB Tech Lab Content Signals project** has published a parallel specification aimed at industry adoption.

Inside the IETF work, separate the *vocabulary* from the *attachment mechanism*. The vocabulary — what `search`

, `ai-input`

and `ai-train`

mean — is alive and was being edited for publication as recently as April 2026 ([draft-ietf-aipref-vocab](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/)). The document that would define how you actually bind those preferences to content over HTTP, [draft-ietf-aipref-attach](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/), **expired on 1 May 2026** and has not been reposted; its text has seen no edits in the working group’s repository since September 2025. So the part of Content Signals a site can act on today — a line in `robots.txt`

— currently rests on the IAB Tech Lab specification and on validator convention, not on a live IETF draft.

That is not a reason to avoid it, but it is a reason to expect the syntax to move. Treat Content Signals as recommended-to-experiment-with, not as a finalised standard. The directive will be ignored by every crawler that does not yet parse it — which today is most of them.

## Why it matters

**A declarative opt-in / opt-out at the right layer.** Existing`User-agent: GPTBot / Disallow:`

directives are coarse — they block a specific bot from fetching anything. Content Signals separates*what*the bot may do from*who*the bot is.**Validators already check for it.**[isitagentready.com](https://isitagentready.com/)explicitly looks for`Content-Signal:`

lines in`robots.txt`

. Sites that want a clean agent-readiness scorecard should add them.**Some major model providers have signalled future support.** Cloudflare, Google, and others have published positions on the IETF drafts.

## How to implement

**Per-group, in robots.txt.** Place the

`Content-Signal:`

line inside the same group as `User-agent:`

and `Allow:`

/ `Disallow:`

.

```
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
```

The example above says: “any crawler may use this content for search, for AI input, and for AI training.” That is the right declaration for a public spec that wants to be readable.

**Different signals per crawler if your policy varies.** Use a targeted group:

```
User-agent: GPTBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no

User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
```

**Pair with crawler-specific blocks where the answer is “no”.** Content Signals is a hint; many crawlers still only obey `Disallow:`

. A `Content-Signal: ai-train=no`

paired with `User-agent: GPTBot \n Disallow: /`

is stronger than either alone.

**Don’t treat it as legal force.** It is a declaration. Compliance is voluntary, and the legal status of “you used my content for training despite my Content-Signal” is still developing.

## Common mistakes

- Spelling:
`Content-Signal:`

(singular). Not`Content-Signals:`

. - Putting the directive outside a group (no preceding
`User-agent:`

line). Some parsers will silently ignore it. - Conflicting with
`Disallow:`

. If you`Disallow: /`

for a bot, that bot was never going to fetch the page to read your`Content-Signal:`

. They contradict; pick one. - Inventing values beyond
`yes`

/`no`

. The vocabulary is small on purpose. - Treating it as a substitute for
`Disallow:`

. It is complementary.

## Verification

`curl -s https://example.com/robots.txt | grep -i content-signal`

lists every directive.[Is It Agent Ready?](https://isitagentready.com/)flips`botAccessControl.contentSignals`

to`pass`

.- Track the
[IETF aipref WG](https://datatracker.ietf.org/wg/aipref/documents/)for the final published vocabulary — names may yet shift before the RFC is published.

## Related topics

## Sources & further reading

[IETF AI Preferences WG (aipref) — drafts](https://datatracker.ietf.org/wg/aipref/documents/)— IETF[IAB Tech Lab — Content Monetization Protocols (CoMP) for AI](https://iabtechlab.com/working-groups/content-monetization-protocols-comp-for-ai-working-group/)— IAB Tech Lab[Is It Agent Ready? — Content Signals check](https://isitagentready.com/)— Is It Agent Ready?[RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)— IETF