# robots.txt patterns that let search crawlers in and keep AI training crawlers out (without wrecking indexing) > Source: > Published: 2026-10-22 08:30:02+00:00 Every few months this request lands in my inbox: *"can we block the AI crawlers but stay in Google?"* The answer is yes, and it takes about 20 lines of `robots.txt` — but only if you stop treating `robots.txt` as one switch. This is the walkthrough I run on WordPress sites. People conflate these constantly. They are different mechanisms with different failure modes. | Decision | Controlled by | If you get it wrong | |---|---|---| | May you **fetch** the page? | `robots.txt` | Page never gets read | | May you **index** the page? | `meta robots` ,`X-Robots-Tag` , canonical | Page appears in results you didn't want | | May you **train on / ground answers in** the content? | Crawler-specific tokens | Content ends up in a corpus you didn't intend | `robots.txt` is a voluntary crawl request. It is not access control. If you need enforcement, that belongs at the server or CDN layer. Keep that in mind before you ship a `Disallow` and assume the problem is solved. Before writing a single line, decide which bucket each token goes in. Write the decision down — future-you will not remember why `PerplexityBot` is allowed. **Bucket A — search index crawlers. Allow.** `Googlebot`, `Bingbot`, `DuckDuckBot`, `Applebot`, `YandexBot`. **Bucket B — training corpora. Usually block.** `GPTBot`, `CCBot` (Common Crawl), `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `Bytespider`. Note that `Google-Extended` and `Applebot-Extended` are **not crawlers**. They're tokens that control whether already-crawled content is used for model training/grounding. Blocking them has no effect on how your pages rank or whether they're indexed — that's exactly why they're the safe ones to disallow. **Bucket C — AI answer surfaces. This is a business call.** `OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, plus user-initiated fetchers like `ChatGPT-User` and `Claude-User`. These fetch pages so an assistant can cite them. Blocking them does not remove you from classic search results. It removes you from being quoted in answers. For a docs site that's a distribution channel; for a paywalled publisher it's a leak. Pick deliberately — the mistake is picking by accident because you copy-pasted a blocklist. ``` # ---- Search: allowed ---- User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / User-agent: YandexBot Allow: / # ---- AI training: blocked ---- User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / # ---- AI answer engines: allowed (flip to Disallow if you disagree) ---- User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / # ---- Fallback for everything else ---- User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /*?replytocom= Disallow: /*?add-to-cart= Sitemap: https://example.com/wp-sitemap.xml ``` **A crawler obeys exactly one group.** It matches the most specific `User-agent` line, and then reads *only* that group. So the moment you add a `Googlebot` group, the `Disallow: /wp-admin/` under `User-agent: *` stops applying to Googlebot. This is the single most common way people break their own site while "just adding AI blocking." If you use specific groups, you have two options: duplicate the shared rules into every group, or leave the shared rules out entirely and keep the specific groups narrow. Don't mix and hope. **`Disallow:` with nothing after it means allow everything.** It is not "block all". `Disallow: /` is block all. **`robots.txt` must return 200 at the root.** A 404 is read by most crawlers as "no restrictions, go ahead". A 5xx tells crawlers something is broken and they may back off crawling. And never disallow `robots.txt` itself. **Blocking a URL does not remove it from the index.** If a page is linked from somewhere crawlable, `Disallow` stops the content being read, not the URL being listed. To remove a page you need `noindex` — and `noindex` only works if the page is *crawlable*. Blocking and de-indexing are opposite operations; running both is how you get "why is this still in Google." Wildcards (`*`, `$`) work on the major search engines. Support elsewhere varies, so treat wildcard lines as best-effort cleanup, not as a security boundary. Read the file back through a parser. Five seconds of Python saves a week of confusion: ``` python import urllib.robotparser from urllib.parse import urljoin BASE = "https://example.com" rp = urllib.robotparser.RobotFileParser() rp.set_url(urljoin(BASE, "/robots.txt")) rp.read() checks = [ ("Googlebot", "/blog/post-1/"), ("Googlebot", "/wp-admin/"), ("GPTBot", "/blog/post-1/"), ("CCBot", "/"), ("OAI-SearchBot", "/blog/post-1/"), ("SomeRandomBot", "/blog/post-1/"), ] for ua, path in checks: allowed = rp.can_fetch(ua, urljoin(BASE, path)) print(f"{ua:15} {path:22} {'ALLOW' if allowed else 'BLOCK'}") ``` Expected shape of the output with the file above: `Googlebot /blog/post-1/` ALLOW, `Googlebot /wp-admin/` **ALLOW** (because the specific group swallowed the `*` rules — that's the bug this test exists to catch), `GPTBot` BLOCK, `OAI-SearchBot` ALLOW. One caveat: Python's `robotparser` doesn't implement `*` and `$` wildcards. Use it for group-matching sanity and check wildcard lines by hand. Then confirm the basics over HTTP: ``` curl -sI https://example.com/robots.txt | head -5 # expect: HTTP/2 200, content-type: text/plain curl -s https://example.com/robots.txt | head -20 ``` Run this every time you touch the file: `/robots.txt` returns 200 with `text/plain` `Googlebot` group exists and contains no `Disallow` on page templates`Disallow: /` sitting in the `*` group`Sitemap:` line points at a URL that returns 200`noindex` pages are crawlable, so the directive can actually be read That last item is the one people skip. Do it on a real URL from each template, not the homepage. `robots.txt` is a request — the log is the truth Once the file is live, the interesting question changes from *"did I write it right"* to *"is anyone honouring it."* You can't answer that from the file; you answer it from request data, per user agent, over time. On WordPress I use [AumCrawl](https://aumcreate.com/plugins/aumcrawl?utm_source=devto&utm_medium=content&utm_campaign=allow-search-block-training) for that: it surfaces which AI crawlers are actually hitting the site and lets you allow or block each one individually at the server level — which is the part `robots.txt` genuinely cannot enforce, since a crawler that ignores the file never reads it in the first place. So the loop is: declare intent in `robots.txt`, enforce it at the server, verify the result in the logs. All three, or you're just writing polite suggestions.