{"slug": "robots-txt-patterns-that-let-search-crawlers-in-and-keep-ai-training-crawlers", "title": "robots.txt patterns that let search crawlers in and keep AI training crawlers out (without wrecking indexing)", "summary": "A developer published a robots.txt walkthrough for WordPress sites that separates search crawlers from AI training crawlers, allowing Googlebot, Bingbot and similar bots while disallowing GPTBot, CCBot, ClaudeBot, Google-Extended, Applebot-Extended and Bytespider. The guide stresses that robots.txt controls fetching only, not indexing or training, and warns that a crawler obeys just one user-agent group, so adding a specific group silently drops the shared wildcard rules like the wp-admin disallow.", "body_md": "Every few months this request lands in my inbox: *\"can we block the AI crawlers but stay in Google?\"* The answer is yes, and it takes about 20 lines of `robots.txt` — but only if you stop treating `robots.txt` as one switch.\n\nThis is the walkthrough I run on WordPress sites.\n\nPeople conflate these constantly. They are different mechanisms with different failure modes.\n\n| Decision | Controlled by | If you get it wrong | \n|---|---|---|\n| May you **fetch** the page? | `robots.txt` | Page never gets read | \n| May you **index** the page? | `meta robots` ,`X-Robots-Tag` , canonical | Page appears in results you didn't want | \n| May you **train on / ground answers in** the content? | Crawler-specific tokens | Content ends up in a corpus you didn't intend | \n\n`robots.txt` is a voluntary crawl request. It is not access control. If you need enforcement, that belongs at the server or CDN layer. Keep that in mind before you ship a `Disallow` and assume the problem is solved.\n\nBefore writing a single line, decide which bucket each token goes in. Write the decision down — future-you will not remember why `PerplexityBot` is allowed.\n\n**Bucket A — search index crawlers. Allow.**\n\n`Googlebot`, `Bingbot`, `DuckDuckBot`, `Applebot`, `YandexBot`.\n\n**Bucket B — training corpora. Usually block.**\n\n`GPTBot`, `CCBot` (Common Crawl), `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `Bytespider`.\n\nNote that `Google-Extended` and `Applebot-Extended` are **not crawlers**. They're tokens that control whether already-crawled content is used for model training/grounding. Blocking them has no effect on how your pages rank or whether they're indexed — that's exactly why they're the safe ones to disallow.\n\n**Bucket C — AI answer surfaces. This is a business call.**\n\n`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, plus user-initiated fetchers like `ChatGPT-User` and `Claude-User`.\n\nThese fetch pages so an assistant can cite them. Blocking them does not remove you from classic search results. It removes you from being quoted in answers. For a docs site that's a distribution channel; for a paywalled publisher it's a leak. Pick deliberately — the mistake is picking by accident because you copy-pasted a blocklist.\n\n```\n# ---- Search: allowed ----\nUser-agent: Googlebot\nAllow: /\n\nUser-agent: Bingbot\nAllow: /\n\nUser-agent: DuckDuckBot\nAllow: /\n\nUser-agent: Applebot\nAllow: /\n\nUser-agent: YandexBot\nAllow: /\n\n# ---- AI training: blocked ----\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: Bytespider\nDisallow: /\n\n# ---- AI answer engines: allowed (flip to Disallow if you disagree) ----\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Claude-SearchBot\nAllow: /\n\n# ---- Fallback for everything else ----\nUser-agent: *\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\nDisallow: /?s=\nDisallow: /*?replytocom=\nDisallow: /*?add-to-cart=\n\nSitemap: https://example.com/wp-sitemap.xml\n```\n\n**A crawler obeys exactly one group.** It matches the most specific `User-agent` line, and then reads *only* that group. So the moment you add a `Googlebot` group, the `Disallow: /wp-admin/` under `User-agent: *` stops applying to Googlebot. This is the single most common way people break their own site while \"just adding AI blocking.\"\n\nIf you use specific groups, you have two options: duplicate the shared rules into every group, or leave the shared rules out entirely and keep the specific groups narrow. Don't mix and hope.\n\n**`Disallow:` with nothing after it means allow everything.** It is not \"block all\". `Disallow: /` is block all.\n\n**`robots.txt` must return 200 at the root.** A 404 is read by most crawlers as \"no restrictions, go ahead\". A 5xx tells crawlers something is broken and they may back off crawling. And never disallow `robots.txt` itself.\n\n**Blocking a URL does not remove it from the index.** If a page is linked from somewhere crawlable, `Disallow` stops the content being read, not the URL being listed. To remove a page you need `noindex` — and `noindex` only works if the page is *crawlable*. Blocking and de-indexing are opposite operations; running both is how you get \"why is this still in Google.\"\n\nWildcards (`*`, `$`) work on the major search engines. Support elsewhere varies, so treat wildcard lines as best-effort cleanup, not as a security boundary.\n\nRead the file back through a parser. Five seconds of Python saves a week of confusion:\n\n``` python\nimport urllib.robotparser\nfrom urllib.parse import urljoin\n\nBASE = \"https://example.com\"\n\nrp = urllib.robotparser.RobotFileParser()\nrp.set_url(urljoin(BASE, \"/robots.txt\"))\nrp.read()\n\nchecks = [\n    (\"Googlebot\", \"/blog/post-1/\"),\n    (\"Googlebot\", \"/wp-admin/\"),\n    (\"GPTBot\", \"/blog/post-1/\"),\n    (\"CCBot\", \"/\"),\n    (\"OAI-SearchBot\", \"/blog/post-1/\"),\n    (\"SomeRandomBot\", \"/blog/post-1/\"),\n]\n\nfor ua, path in checks:\n    allowed = rp.can_fetch(ua, urljoin(BASE, path))\n    print(f\"{ua:15} {path:22} {'ALLOW' if allowed else 'BLOCK'}\")\n```\n\nExpected shape of the output with the file above: `Googlebot /blog/post-1/` ALLOW, `Googlebot /wp-admin/` **ALLOW** (because the specific group swallowed the `*` rules — that's the bug this test exists to catch), `GPTBot` BLOCK, `OAI-SearchBot` ALLOW.\n\nOne caveat: Python's `robotparser` doesn't implement `*` and `$` wildcards. Use it for group-matching sanity and check wildcard lines by hand.\n\nThen confirm the basics over HTTP:\n\n```\ncurl -sI https://example.com/robots.txt | head -5\n# expect: HTTP/2 200, content-type: text/plain\n\ncurl -s https://example.com/robots.txt | head -20\n```\n\nRun this every time you touch the file:\n\n`/robots.txt` returns 200 with `text/plain`\n`Googlebot` group exists and contains no `Disallow` on page templates`Disallow: /` sitting in the `*` group`Sitemap:` line points at a URL that returns 200`noindex` pages are crawlable, so the directive can actually be read\nThat last item is the one people skip. Do it on a real URL from each template, not the homepage.\n\n`robots.txt` is a request — the log is the truth\nOnce the file is live, the interesting question changes from *\"did I write it right\"* to *\"is anyone honouring it.\"* You can't answer that from the file; you answer it from request data, per user agent, over time.\n\nOn WordPress I use [AumCrawl](https://aumcreate.com/plugins/aumcrawl?utm_source=devto&utm_medium=content&utm_campaign=allow-search-block-training) for that: it surfaces which AI crawlers are actually hitting the site and lets you allow or block each one individually at the server level — which is the part `robots.txt` genuinely cannot enforce, since a crawler that ignores the file never reads it in the first place.\n\nSo the loop is: declare intent in `robots.txt`, enforce it at the server, verify the result in the logs. All three, or you're just writing polite suggestions.", "url": "https://wpnews.pro/news/robots-txt-patterns-that-let-search-crawlers-in-and-keep-ai-training-crawlers", "canonical_source": "https://dev.to/aumcreate/robotstxt-patterns-that-let-search-crawlers-in-and-keep-ai-training-crawlers-out-without-wrecking-2oko", "published_at": "2026-10-22 08:30:02+00:00", "updated_at": "2026-10-22 08:40:30.240441+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "structured-data", "developer-tools"], "entities": ["Googlebot", "Bingbot", "GPTBot", "CCBot", "ClaudeBot", "Google-Extended", "PerplexityBot", "WordPress"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/robots-txt-patterns-that-let-search-crawlers-in-and-keep-ai-training-crawlers", "markdown": "https://wpnews.pro/news/robots-txt-patterns-that-let-search-crawlers-in-and-keep-ai-training-crawlers.md", "text": "https://wpnews.pro/news/robots-txt-patterns-that-let-search-crawlers-in-and-keep-ai-training-crawlers.txt", "jsonld": "https://wpnews.pro/news/robots-txt-patterns-that-let-search-crawlers-in-and-keep-ai-training-crawlers.jsonld"}}