Your robots.txt may express an AI policy you did not write.
We checked the homepage and robots.txt of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample.
Treat AI crawler rules as deployment configuration. Review them when you change hosting, enable a CDN feature, adopt a starter template, or hand site operations to someone else.
Read the full research and methodology.
A common configuration blocks model training while keeping a site available in AI-assisted search and browsing:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
These are separate crawlers with separate purposes. In our sample, 839 of the 945 sites that block GPTBot, or 88.8%, still allow OAI-SearchBot. That is a deliberate and useful distinction if your goal is to opt out of training while remaining eligible to be cited in ChatGPT search.
The same pattern appears across AI labs. ClaudeBot is explicitly blocked by 10.1% of the 9,037 tools, while Claude-SearchBot is blocked by just 0.1%. Google-Extended is blocked by 9.9%, but its purpose is also distinct from ordinary Google Search crawling.
Do not assume a broad-looking rule has the result you want. Check the actual crawler names and decide which capabilities you want to permit.
Start by opening the public URL:
https://your-domain.example/robots.txt
Then look for three things:
User-agent: GPTBot. Disallow: / directly inside that group.User-agent: *, that could affect all crawlers.
Our measurement only counts a site as blocking GPTBot when the named GPTBot group itself contains Disallow: /. This matters because ordinary technical exclusions are widespread. Only 31 sites in the 9,037-site sample, or 0.3%, block every crawler outright. Meanwhile, 44% have a path-level Disallow rule in a wildcard group, often for an admin area or API route. That is normal site hygiene, not necessarily an AI policy.
Keep sensitive paths protected through authentication and application controls too. robots.txt is a crawler instruction, not access control.
Hosting correlated strongly with whether a tool blocked GPTBot. Of 3,749 Cloudflare-hosted sites, 22.4% explicitly blocked it. On Vercel, 5.2% of 2,556 sites did. Netlify was 4.0% across 273 sites, Fastly 3.9% across 361, and CloudFront 3.4% across 328.
That makes Cloudflare-hosted tools more than four times as likely to block GPTBot as Vercel-hosted tools in this sample.
The likely explanation is operational rather than philosophical. A hosting control that offers to block AI bots can write or manage crawler policy at scale. That is convenient, but it can make a consequential content policy feel like a harmless performance or security toggle.
Whenever you enable one of these settings, verify the deployed result rather than relying on the dashboard label. Fetch robots.txt after deployment and include it in your release checklist.
A small policy file is easier to maintain than an inherited one. If you want to block training but preserve discoverability, state that clearly with named groups. If you want to allow everything, avoid leaving old template rules behind. If you want a complete block, understand that wildcard rules can affect conventional search crawlers too.
The broader data shows this is becoming more common: 13.6% of 2,900 domains registered in 2026 block GPTBot, compared with 6.9% of 145 domains registered in 2023. That makes it even more important to know whether your policy is intentional.
For the complete crawler breakdown, hosting results, and methodology, see How many AI tools block GPTBot.