{"slug": "gptbot-in-robots-txt-the-hosting-toggle-developers-need-to-check", "title": "GPTBot in robots.txt: the hosting toggle developers need to check", "summary": "A study of 9,037 AI tools found that 10.5% explicitly block OpenAI's GPTBot in their robots.txt files, with Cloudflare-hosted sites more than four times as likely to block it as Vercel-hosted sites. The research highlights that hosting controls can inadvertently set AI crawler policies, urging developers to verify their robots.txt after deployment.", "body_md": "Your `robots.txt` may express an AI policy you did not write.\n\nWe checked the homepage and `robots.txt` of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample.\n\nTreat AI crawler rules as deployment configuration. Review them when you change hosting, enable a CDN feature, adopt a starter template, or hand site operations to someone else.\n\nRead the [full research and methodology](https://www.directree.io/research/ai-tools-blocking-gptbot).\n\nA common configuration blocks model training while keeping a site available in AI-assisted search and browsing:\n\n```\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n```\n\nThese are separate crawlers with separate purposes. In our sample, 839 of the 945 sites that block GPTBot, or 88.8%, still allow OAI-SearchBot. That is a deliberate and useful distinction if your goal is to opt out of training while remaining eligible to be cited in ChatGPT search.\n\nThe same pattern appears across AI labs. ClaudeBot is explicitly blocked by 10.1% of the 9,037 tools, while Claude-SearchBot is blocked by just 0.1%. Google-Extended is blocked by 9.9%, but its purpose is also distinct from ordinary Google Search crawling.\n\nDo not assume a broad-looking rule has the result you want. Check the actual crawler names and decide which capabilities you want to permit.\n\nStart by opening the public URL:\n\n```\nhttps://your-domain.example/robots.txt\n```\n\nThen look for three things:\n\n`User-agent: GPTBot`.` Disallow: /` directly inside that group.`User-agent: *`, that could affect all crawlers.\nOur measurement only counts a site as blocking GPTBot when the named GPTBot group itself contains `Disallow: /`. This matters because ordinary technical exclusions are widespread. Only 31 sites in the 9,037-site sample, or 0.3%, block every crawler outright. Meanwhile, 44% have a path-level `Disallow` rule in a wildcard group, often for an admin area or API route. That is normal site hygiene, not necessarily an AI policy.\n\nKeep sensitive paths protected through authentication and application controls too. `robots.txt` is a crawler instruction, not access control.\n\nHosting correlated strongly with whether a tool blocked GPTBot. Of 3,749 Cloudflare-hosted sites, 22.4% explicitly blocked it. On Vercel, 5.2% of 2,556 sites did. Netlify was 4.0% across 273 sites, Fastly 3.9% across 361, and CloudFront 3.4% across 328.\n\nThat makes Cloudflare-hosted tools more than four times as likely to block GPTBot as Vercel-hosted tools in this sample.\n\nThe likely explanation is operational rather than philosophical. A hosting control that offers to block AI bots can write or manage crawler policy at scale. That is convenient, but it can make a consequential content policy feel like a harmless performance or security toggle.\n\nWhenever you enable one of these settings, verify the deployed result rather than relying on the dashboard label. Fetch `robots.txt` after deployment and include it in your release checklist.\n\nA small policy file is easier to maintain than an inherited one. If you want to block training but preserve discoverability, state that clearly with named groups. If you want to allow everything, avoid leaving old template rules behind. If you want a complete block, understand that wildcard rules can affect conventional search crawlers too.\n\nThe broader data shows this is becoming more common: 13.6% of 2,900 domains registered in 2026 block GPTBot, compared with 6.9% of 145 domains registered in 2023. That makes it even more important to know whether your policy is intentional.\n\nFor the complete crawler breakdown, hosting results, and methodology, see [How many AI tools block GPTBot](https://www.directree.io/research/ai-tools-blocking-gptbot).", "url": "https://wpnews.pro/news/gptbot-in-robots-txt-the-hosting-toggle-developers-need-to-check", "canonical_source": "https://dev.to/directree/gptbot-in-robotstxt-the-hosting-toggle-developers-need-to-check-10he", "published_at": "2026-09-07 09:35:38+00:00", "updated_at": "2026-09-07 09:59:34.680862+00:00", "lang": "en", "topics": ["ai-policy", "ai-tools", "developer-tools"], "entities": ["OpenAI", "GPTBot", "Cloudflare", "Vercel", "Netlify", "Fastly", "CloudFront", "directree"], "alternates": {"html": "https://wpnews.pro/news/gptbot-in-robots-txt-the-hosting-toggle-developers-need-to-check", "markdown": "https://wpnews.pro/news/gptbot-in-robots-txt-the-hosting-toggle-developers-need-to-check.md", "text": "https://wpnews.pro/news/gptbot-in-robots-txt-the-hosting-toggle-developers-need-to-check.txt", "jsonld": "https://wpnews.pro/news/gptbot-in-robots-txt-the-hosting-toggle-developers-need-to-check.jsonld"}}