robots.txt patterns that let search crawlers in and keep AI training crawlers out (without wrecking indexing)

A developer published a robots.txt walkthrough for WordPress sites that separates search crawlers from AI training crawlers, allowing Googlebot, Bingbot and similar bots while disallowing GPTBot, CCBot, ClaudeBot, Google-Extended, Applebot-Extended and Bytespider. The guide stresses that robots.txt controls fetching only, not indexing or training, and warns that a crawler obeys just one user-agent group, so adding a specific group silently drops the shared wildcard rules like the wp-admin disallow.

Every few months this request lands in my inbox: "can we block the AI crawlers but stay in Google?" The answer is yes, and it takes about 20 lines of robots.txt — but only if you stop treating robots.txt as one switch. This is the walkthrough I run on WordPress sites. People conflate these constantly. They are different mechanisms with different failure modes. | Decision | Controlled by | If you get it wrong | |---|---|---| | May you fetch the page? | robots.txt | Page never gets read | | May you index the page? | meta robots , X-Robots-Tag , canonical | Page appears in results you didn't want | | May you train on / ground answers in the content? | Crawler-specific tokens | Content ends up in a corpus you didn't intend | robots.txt is a voluntary crawl request. It is not access control. If you need enforcement, that belongs at the server or CDN layer. Keep that in mind before you ship a Disallow and assume the problem is solved. Before writing a single line, decide which bucket each token goes in. Write the decision down — future-you will not remember why PerplexityBot is allowed. Bucket A — search index crawlers. Allow. Googlebot , Bingbot , DuckDuckBot , Applebot , YandexBot . Bucket B — training corpora. Usually block. GPTBot , CCBot Common Crawl , ClaudeBot , Google-Extended , Applebot-Extended , Bytespider . Note that Google-Extended and Applebot-Extended are not crawlers . They're tokens that control whether already-crawled content is used for model training/grounding. Blocking them has no effect on how your pages rank or whether they're indexed — that's exactly why they're the safe ones to disallow. Bucket C — AI answer surfaces. This is a business call. OAI-SearchBot , PerplexityBot , Claude-SearchBot , plus user-initiated fetchers like ChatGPT-User and Claude-User . These fetch pages so an assistant can cite them. Blocking them does not remove you from classic search results. It removes you from being quoted in answers. For a docs site that's a distribution channel; for a paywalled publisher it's a leak. Pick deliberately — the mistake is picking by accident because you copy-pasted a blocklist. ---- Search: allowed ---- User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / User-agent: YandexBot Allow: / ---- AI training: blocked ---- User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / ---- AI answer engines: allowed flip to Disallow if you disagree ---- User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / ---- Fallback for everything else ---- User-agent: Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: / ?replytocom= Disallow: / ?add-to-cart= Sitemap: https://example.com/wp-sitemap.xml A crawler obeys exactly one group. It matches the most specific User-agent line, and then reads only that group. So the moment you add a Googlebot group, the Disallow: /wp-admin/ under User-agent: stops applying to Googlebot. This is the single most common way people break their own site while "just adding AI blocking." If you use specific groups, you have two options: duplicate the shared rules into every group, or leave the shared rules out entirely and keep the specific groups narrow. Don't mix and hope. Disallow: with nothing after it means allow everything. It is not "block all". Disallow: / is block all. robots.txt must return 200 at the root. A 404 is read by most crawlers as "no restrictions, go ahead". A 5xx tells crawlers something is broken and they may back off crawling. And never disallow robots.txt itself. Blocking a URL does not remove it from the index. If a page is linked from somewhere crawlable, Disallow stops the content being read, not the URL being listed. To remove a page you need noindex — and noindex only works if the page is crawlable . Blocking and de-indexing are opposite operations; running both is how you get "why is this still in Google." Wildcards , $ work on the major search engines. Support elsewhere varies, so treat wildcard lines as best-effort cleanup, not as a security boundary. Read the file back through a parser. Five seconds of Python saves a week of confusion: python import urllib.robotparser from urllib.parse import urljoin BASE = "https://example.com" rp = urllib.robotparser.RobotFileParser rp.set url urljoin BASE, "/robots.txt" rp.read checks = "Googlebot", "/blog/post-1/" , "Googlebot", "/wp-admin/" , "GPTBot", "/blog/post-1/" , "CCBot", "/" , "OAI-SearchBot", "/blog/post-1/" , "SomeRandomBot", "/blog/post-1/" , for ua, path in checks: allowed = rp.can fetch ua, urljoin BASE, path print f"{ua:15} {path:22} {'ALLOW' if allowed else 'BLOCK'}" Expected shape of the output with the file above: Googlebot /blog/post-1/ ALLOW, Googlebot /wp-admin/ ALLOW because the specific group swallowed the rules — that's the bug this test exists to catch , GPTBot BLOCK, OAI-SearchBot ALLOW. One caveat: Python's robotparser doesn't implement and $ wildcards. Use it for group-matching sanity and check wildcard lines by hand. Then confirm the basics over HTTP: curl -sI https://example.com/robots.txt | head -5 expect: HTTP/2 200, content-type: text/plain curl -s https://example.com/robots.txt | head -20 Run this every time you touch the file: /robots.txt returns 200 with text/plain Googlebot group exists and contains no Disallow on page templates Disallow: / sitting in the group Sitemap: line points at a URL that returns 200 noindex pages are crawlable, so the directive can actually be read That last item is the one people skip. Do it on a real URL from each template, not the homepage. robots.txt is a request — the log is the truth Once the file is live, the interesting question changes from "did I write it right" to "is anyone honouring it." You can't answer that from the file; you answer it from request data, per user agent, over time. On WordPress I use AumCrawl https://aumcreate.com/plugins/aumcrawl?utm source=devto&utm medium=content&utm campaign=allow-search-block-training for that: it surfaces which AI crawlers are actually hitting the site and lets you allow or block each one individually at the server level — which is the part robots.txt genuinely cannot enforce, since a crawler that ignores the file never reads it in the first place. So the loop is: declare intent in robots.txt , enforce it at the server, verify the result in the logs. All three, or you're just writing polite suggestions.