{"slug": "ai-crawlers-list-training-vs-search-user-agents", "title": "AI Crawlers List: Training vs. Search User Agents", "summary": "Sitecheckly surveyed the Tranco top 10,000 domains (list 7462X) in November 2026, analyzing 4,934 sites and finding that 9.2% of sites with a readable robots.txt blocked AI training crawlers while still allowing AI search crawlers. The survey distinguishes training user agents such as GPTBot, ClaudeBot and Google-Extended from search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot, noting that robots.txt rules may not apply to user-initiated fetches like ChatGPT-User.", "body_md": "*Originally published on [Sitecheckly](https://sitecheckly.com/blog/ai-crawlers-list-user-agents).*\n\nAI crawlers serve different purposes. Blocking [GPTBot targets training access, while blocking OAI-SearchBot affects ChatGPT search](https://platform.openai.com/docs/bots). The same distinction applies to [ClaudeBot and Claude-SearchBot](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web). [Google-Extended controls Gemini training, not Google Search or AI Overviews](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers). Choose robots.txt rules by purpose rather than blocking every AI-related user agent.\n\nIn the survey, 9.2% of sites blocked training crawlers while still allowing AI search crawlers. That is a useful distinction if you want to limit training access without closing off search discovery.\n\nThe November 2026 Sitecheckly survey covered the Tranco top 10,000, using Tranco list 7462X. We could analyze 4,934 of 10,000 domains. The rest were mostly CDN, DNS and API hosts, sites that ask crawlers to stay away in robots.txt, or sites that blocked automated requests. The training-versus-search finding uses the 4,546 sites with a readable robots.txt. See the [survey methodology](https://sitecheckly.com/methodology).\n\nTreat your crawler policy as separate decisions:\n\nA training opt-out is not the same as a search opt-out. Start with the outcome you want, then choose the matching user-agent groups.\n\nThis AI crawlers list separates training, search and user-requested fetching. Google-Extended is included because it belongs in the policy decision, even though it is a control token rather than a separate fetching crawler.\n\nThe allow and block columns describe the whole-site patterns shown below. They are alternatives, not instructions to apply both.\n\n| User-agent token | Operator | Purpose | Allow rule | Block rule | \n|---|---|---|---|---|\n| GPTBot | OpenAI | Model training | Allow all paths | Disallow all paths | \n| OAI-SearchBot | OpenAI | ChatGPT search | Allow all paths | Disallow all paths | \n| ChatGPT-User | OpenAI | Fetching on a user's request | Allow all paths* | Disallow all paths* | \n| ClaudeBot | Anthropic | Model training | Allow all paths | Disallow all paths | \n| Claude-SearchBot | Anthropic | Search | Allow all paths | Disallow all paths | \n| PerplexityBot | Perplexity | Search | Allow all paths | Disallow all paths | \n| Google-Extended |  | Gemini training control, not a separate crawler | Allow all paths | Disallow all paths | \n\nFor **OAI-SearchBot vs GPTBot**, the key difference is search versus training. ChatGPT-User is separate from both: it fetches content in response to a user's request. *OpenAI notes that robots.txt rules may not apply to these user-initiated requests, so a ChatGPT-User block is not a guaranteed access restriction.* See [OpenAI's bot documentation](https://platform.openai.com/docs/bots).\n\nFor Anthropic, use the distinct tokens for distinct decisions: [ClaudeBot is for training, while Claude-SearchBot supports search](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web).\n\n[PerplexityBot is Perplexity's search crawler](https://docs.perplexity.ai/guides/bots). Do not classify it as a training crawler just because it is associated with an AI service.\n\nUse a user-agent group to name the crawler, then add the path rule. In these examples, the root path covers the whole site under that robots.txt policy. The group and path matching rules come from [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309).\n\nTo block AI training access through GPTBot:\n\n```\nUser-agent: GPTBot\nDisallow: /\n```\n\nTo allow AI search access through OAI-SearchBot:\n\n```\nUser-agent: OAI-SearchBot\nAllow: /\n```\n\nThese groups can coexist. Blocking GPTBot does not require blocking OAI-SearchBot.\n\nEach group below expresses an allow-all policy for its named token. Select the groups that match your intended policy. The ChatGPT-User caveat above still applies.\n\n```\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: Claude-SearchBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /\n```\n\nThese are the equivalent disallow-all groups. This is a reference, not a recommended file to paste wholesale. Blocking every listed search crawler would work against a policy intended to preserve AI search access.\n\n```\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: OAI-SearchBot\nDisallow: /\n\nUser-agent: ChatGPT-User\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Claude-SearchBot\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n```\n\nFor a training-blocked, search-allowed policy, select the block groups for GPTBot, ClaudeBot and Google-Extended. Select the allow groups for OAI-SearchBot, Claude-SearchBot and PerplexityBot. Make the user-requested access decision separately.\n\nDo not assume that the last line in the file wins. Under [RFC 9309's matching rules](https://datatracker.ietf.org/doc/html/rfc9309):\n\nFor example, this group blocks GPTBot from the site except for the more specific public path:\n\n```\nUser-agent: GPTBot\nDisallow: /\nAllow: /public/\n```\n\nAn allow rule is permission under the robots.txt policy. It is not a promise that a service will crawl, use or cite the page. Keep access eligibility separate from the goal of earning a citation.\n\nNo. To opt out of Gemini training through this control, use:\n\n```\nUser-agent: Google-Extended\nDisallow: /\n```\n\nGoogle-Extended is a robots.txt control token. It is not a separate crawler making requests under that name. Blocking it does not block Google Search or AI Overviews; Googlebot is the relevant Search crawler. See [Google's crawler documentation](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers).\n\nDo not substitute a Googlebot block when your intended decision is only about Gemini training. Those controls address different purposes.\n\nStart with the published robots.txt, not just the copy in your editor. Compare its user-agent groups with the policy you meant to publish.\n\nUse this checklist:\n\nThen run the free, no-sign-up [Sitecheckly AEO audit](https://sitecheckly.com/aeo-audit) on the page URL. Each run audits one page: the URL you enter, not the whole site.\n\nFor context beyond crawler access, read the [answer engine optimization explainer](https://sitecheckly.com/blog/what-is-answer-engine-optimization). Use it to separate the access question from the broader work of making a page useful in search answers.\n\n**What to do next**\n\n**Is ChatGPT-User the same as OAI-SearchBot?**\n\nNo. ChatGPT-User fetches content in response to a user's request; OAI-SearchBot supports ChatGPT search. See [OpenAI's bot documentation](https://platform.openai.com/docs/bots).\n\n**Does allowing an AI search crawler guarantee a citation?**\n\nNo. An allow rule permits crawling under that robots.txt policy; it does not guarantee that a service will crawl, use or cite the page.\n\n**What happens if two robots.txt path rules conflict?**\n\nThe most specific matching path wins. If equivalent allow and disallow rules match, the protocol says the allow rule should be used. See [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309).\n\nWant to see how your own site does? Run a free audit at [sitecheckly.com](https://sitecheckly.com/?utm_source=devto).\n\n*Figures in this article come from Sitecheckly's own audit data; see the [methodology](https://sitecheckly.com/methodology).*", "url": "https://wpnews.pro/news/ai-crawlers-list-training-vs-search-user-agents", "canonical_source": "https://dev.to/hieu_le_f3630aa13143b35d8/ai-crawlers-list-training-vs-search-user-agents-1p9h", "published_at": "2026-11-03 09:41:24+00:00", "updated_at": "2026-11-03 10:03:42.538298+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "artificial-intelligence"], "entities": ["Sitecheckly", "OpenAI", "Anthropic", "Perplexity", "Google", "GPTBot", "OAI-SearchBot", "ClaudeBot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-crawlers-list-training-vs-search-user-agents", "markdown": "https://wpnews.pro/news/ai-crawlers-list-training-vs-search-user-agents.md", "text": "https://wpnews.pro/news/ai-crawlers-list-training-vs-search-user-agents.txt", "jsonld": "https://wpnews.pro/news/ai-crawlers-list-training-vs-search-user-agents.jsonld"}}