Major AI vendors publish named user-agents for their crawlers. Setting an explicit allow or disallow per agent is the clearest way to control how your content is used.
Adoption: 13.059% of 668,523 websites in the HTTP Archive September 2026 desktop sample
What it is #
The robots.txt file (see robots.txt) accepts rules per user-agent. Every major AI vendor now publishes a named user-agent for its training and retrieval crawlers, so you can allow or block each one independently of search.
The big ones, as of 2026:
- GPTBot — OpenAI training crawler.
- OAI-SearchBot — OpenAI retrieval crawler used by ChatGPT browsing.
- ChatGPT-User — on-demand fetches when a ChatGPT user asks for a URL.
- ClaudeBot — Anthropic training crawler.
- Claude-SearchBot — Anthropic search-indexing crawler.
- Claude-User — on-demand fetches when a Claude user’s question needs a page.
- anthropic-ai — legacy Anthropic user-agent, still seen.
- Google-Extended — opts out of Gemini and Vertex training without affecting Search.
- Applebot-Extended — opts out of Apple Intelligence training without affecting Siri/Spotlight.
- PerplexityBot — Perplexity retrieval crawler.
- Bytespider — ByteDance crawler, widely used for training.
- CCBot — Common Crawl, the dataset behind many open models.
Why it matters #
A blanket Disallow: / blocks everyone, including search. Naming agents lets you make precise decisions: allow retrieval bots so your content can be cited live, block training bots if you do not want to feed model weights, or the reverse.
Compliance is honour-based. Reputable vendors document and respect their user-agents. Unidentified scrapers will ignore robots.txt; defend against those with rate limits or WAF rules, not robots.
How to implement #
A reasonable default that allows search and retrieval but opts out of training:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap_index.xml
Rules of thumb:
- Read each vendor’s documentation; user-agents change.
- Decisions are per-host. A subdomain needs its own robots.txt.
- Combine with /llms.txt when you want to allow access but steer agents to specific pages.
- For paywalled or member content, keep auth in place — robots.txt is not a security boundary.
A shared training preference
An Internet-Draft published on 30 October 2026 defines one name for stating a training preference to every provider that honours it, instead of listing each vendor:
User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
This looks like a crawler rule, but it is not one. No bot needs to send that name, and the group does not itself stop anything being fetched. Disallow states a preference against generative AI training on those paths, and Allow states a preference in favour. To limit crawling, keep rules for each crawler’s own user-agent. A User-agent: * group does not count, so a blanket Disallow: / under * expresses no preference through this mechanism, and a file without the group expresses none at all.
Nothing has been announced as live yet. Microsoft will honour it from 1 February 2027 and Google intends to in early 2027. OpenAI aims to in the coming months, and Anthropic has started implementing it. The draft is an individual submission, not yet a working-group document, so the details may change.
Keep your per-vendor rules, because the shared group sits alongside them and does not replace them. A vendor’s own token can take precedence: Google applies the shared opt-out only where Google-Extended is not explicitly specified.
This site does not publish the group: it has no training opt-out to express, and it already allows these crawlers by name.
Common mistakes #
- Blocking
*and forgetting that this also blocks Google. - Blocking only
GPTBotand assuming no OpenAI surface can see your site —OAI-SearchBotandChatGPT-Userare separate. The same split applies to Anthropic:ClaudeBot,Claude-SearchBotandClaude-Userare governed independently. - Leaving stale user-agents from 2023 lists. Names change; the vendor docs are the source of truth.
- Treating disallow as deletion. Content already in training sets stays there.
- Using
User-agent: __usage_ai-trainingto keep a crawler out. It states a preference about training, not a crawling rule.
Verification #
- Fetch
https://example.com/robots.txtand check each rule renders. - Cross-reference your list against the linked vendor pages — they are updated regularly.
- Watch server logs for the user-agents you care about and confirm they back off after a rule change.
Related topics #
Sources & further reading #
- OpenAI — GPTBot — OpenAI
- Anthropic — Crawlers — Anthropic
- Google — Google-Extended — Google Crawling Infrastructure
- Apple — Applebot-Extended — Apple
- Machine-Readable Training Preference Signal (Internet-Draft) — IETF