09:35
2026-09-07
dev.to
ai-policy
GPTBot in robots.txt: the hosting toggle developers need to check
A study of 9,037 AI tools found that 10.5% explicitly block OpenAI's GPTBot in their robots.txt files, with Cloudflare-hosted sites more than four times as likely to block it as Vercel-hosted sites. T…