Every few months this request lands in my inbox: "can we block the AI crawlers but stay in Google?" The answer is yes, and it takes about 20 lines of robots.txt — but only if you stop treating robots.txt as one switch.
This is the walkthrough I run on WordPress sites.
People conflate these constantly. They are different mechanisms with different failure modes.
| Decision | Controlled by | If you get it wrong |
|---|---|---|
| May you fetch the page? | robots.txt |
Page never gets read |
| May you index the page? | meta robots ,X-Robots-Tag , canonical |
Page appears in results you didn't want |
| May you train on / ground answers in the content? | Crawler-specific tokens | Content ends up in a corpus you didn't intend |
robots.txt is a voluntary crawl request. It is not access control. If you need enforcement, that belongs at the server or CDN layer. Keep that in mind before you ship a Disallow and assume the problem is solved.
Before writing a single line, decide which bucket each token goes in. Write the decision down — future-you will not remember why PerplexityBot is allowed.
Bucket A — search index crawlers. Allow.
Googlebot, Bingbot, DuckDuckBot, Applebot, YandexBot.
Bucket B — training corpora. Usually block.
GPTBot, CCBot (Common Crawl), ClaudeBot, Google-Extended, Applebot-Extended, Bytespider.
Note that Google-Extended and Applebot-Extended are not crawlers. They're tokens that control whether already-crawled content is used for model training/grounding. Blocking them has no effect on how your pages rank or whether they're indexed — that's exactly why they're the safe ones to disallow.
Bucket C — AI answer surfaces. This is a business call.
OAI-SearchBot, PerplexityBot, Claude-SearchBot, plus user-initiated fetchers like ChatGPT-User and Claude-User.
These fetch pages so an assistant can cite them. Blocking them does not remove you from classic search results. It removes you from being quoted in answers. For a docs site that's a distribution channel; for a paywalled publisher it's a leak. Pick deliberately — the mistake is picking by accident because you copy-pasted a blocklist.
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: DuckDuckBot
Allow: /
User-agent: Applebot
Allow: /
User-agent: YandexBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /*?replytocom=
Disallow: /*?add-to-cart=
Sitemap: https://example.com/wp-sitemap.xml
A crawler obeys exactly one group. It matches the most specific User-agent line, and then reads only that group. So the moment you add a Googlebot group, the Disallow: /wp-admin/ under User-agent: * stops applying to Googlebot. This is the single most common way people break their own site while "just adding AI blocking."
If you use specific groups, you have two options: duplicate the shared rules into every group, or leave the shared rules out entirely and keep the specific groups narrow. Don't mix and hope.
Disallow: with nothing after it means allow everything. It is not "block all". Disallow: / is block all.
robots.txt must return 200 at the root. A 404 is read by most crawlers as "no restrictions, go ahead". A 5xx tells crawlers something is broken and they may back off crawling. And never disallow robots.txt itself.
Blocking a URL does not remove it from the index. If a page is linked from somewhere crawlable, Disallow stops the content being read, not the URL being listed. To remove a page you need noindex — and noindex only works if the page is crawlable. Blocking and de-indexing are opposite operations; running both is how you get "why is this still in Google."
Wildcards (*, $) work on the major search engines. Support elsewhere varies, so treat wildcard lines as best-effort cleanup, not as a security boundary.
Read the file back through a parser. Five seconds of Python saves a week of confusion:
import urllib.robotparser
from urllib.parse import urljoin
BASE = "https://example.com"
rp = urllib.robotparser.RobotFileParser()
rp.set_url(urljoin(BASE, "/robots.txt"))
rp.read()
checks = [
("Googlebot", "/blog/post-1/"),
("Googlebot", "/wp-admin/"),
("GPTBot", "/blog/post-1/"),
("CCBot", "/"),
("OAI-SearchBot", "/blog/post-1/"),
("SomeRandomBot", "/blog/post-1/"),
]
for ua, path in checks:
allowed = rp.can_fetch(ua, urljoin(BASE, path))
print(f"{ua:15} {path:22} {'ALLOW' if allowed else 'BLOCK'}")
Expected shape of the output with the file above: Googlebot /blog/post-1/ ALLOW, Googlebot /wp-admin/ ALLOW (because the specific group swallowed the * rules — that's the bug this test exists to catch), GPTBot BLOCK, OAI-SearchBot ALLOW.
One caveat: Python's robotparser doesn't implement * and $ wildcards. Use it for group-matching sanity and check wildcard lines by hand.
Then confirm the basics over HTTP:
curl -sI https://example.com/robots.txt | head -5
curl -s https://example.com/robots.txt | head -20
Run this every time you touch the file:
/robots.txt returns 200 with text/plain
Googlebot group exists and contains no Disallow on page templatesDisallow: / sitting in the * groupSitemap: line points at a URL that returns 200noindex pages are crawlable, so the directive can actually be read
That last item is the one people skip. Do it on a real URL from each template, not the homepage.
robots.txt is a request — the log is the truth
Once the file is live, the interesting question changes from "did I write it right" to "is anyone honouring it." You can't answer that from the file; you answer it from request data, per user agent, over time.
On WordPress I use AumCrawl for that: it surfaces which AI crawlers are actually hitting the site and lets you allow or block each one individually at the server level — which is the part robots.txt genuinely cannot enforce, since a crawler that ignores the file never reads it in the first place.
So the loop is: declare intent in robots.txt, enforce it at the server, verify the result in the logs. All three, or you're just writing polite suggestions.