GPTBot and OAI-SearchBot are two different OpenAI crawlers. GPTBot collects content that may be used for model training. OAI-SearchBot finds pages for ChatGPT search. Blocking one doesn't block the other, and mixing them up is an easy robots.txt mistake to make.
The same split exists at Anthropic and Perplexity. If the wrong bot is blocked, a site can disappear from AI search answers while still ranking fine on Google, because Google uses its own crawlers.
And this isn't just an SEO problem. AI visibility agency RankingBite calls it the "Distribution" layer of AI visibility: being present in the sources AI models actually learn from and retrieve. A single Disallow line can break that layer before content quality even comes into play.
So here's what each bot does, the robots.txt setup I'd recommend, and a few commands to check it actually works.
How to Configure robots.txt for AI Crawlers
For most product sites, SaaS and B2B brands, this is the setup I'd start with: let AI search in, and make a separate call on training. This one allows search and user fetches, and blocks training: A few things that are easy to miss:
• Specific rules win. A crawler follows the group that names it, not the User-agent: * group. An old Disallow: / under * will not block OAI-SearchBot if it has its own Allow group.
• Keep private paths private. Add Disallow: /admin/ or similar inside each AI group too, since each group is read on its own.
• Give it time. OpenAI and Perplexity both say changes can take about 24 hours to take effect.
• User bots are different. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person triggers the visit.
Check the CDN and Firewall, Not Just robots.txt
robots.txt is a request, not a gate. A CDN or web application firewall can block a bot before it ever reads the file.
If the site is on Cloudflare, check there first. Since July 1, 2025, it blocks AI crawlers by default for new customers, and existing customers can turn blocking on with one click. Its AI Crawl Control dashboard, available on every plan, shows which AI crawlers hit the site and lets you allow or deny each one. Other CDNs and WAFs, plus security plugins on platforms like WordPress, can have similar bot rules. How to Verify AI Bots Can Reach the Site
Don't guess, check the logs. On an Nginx or Apache server, something like this shows recent AI bot hits and their status codes:
This prints the status code and full user agent for each hit, assuming the default combined log format. If the server uses a custom format, adjust the awk columns to match. What to look for:
• 200 responses: the bot is getting through.
• 403 or 429 responses: something is blocking or rate-limiting it, usually the CDN or WAF.
• No hits at all: the bot may be blocked before it reaches the origin, or it has not visited yet.
One catch: user-agent strings are easy to fake, so don't trust the name alone. OpenAI and Perplexity publish official IP ranges for their bots:
• [https://openai.com/gptbot.json](https://openai.com/gptbot.json)
• [https://openai.com/searchbot.json](https://openai.com/searchbot.json)
• [https://openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json)
• [https://www.perplexity.com/perplexitybot.json](https://www.perplexity.com/perplexitybot.json)
Compare the source IPs in the logs against these lists. Anthropic advises against blocking its bots by IP, and recommends robots.txt instead.
A quick curl -A "OAI-SearchBot" -I https://example.com test can show whether a simple user-agent rule blocks the bot. It will not catch CDN rules that verify bots by IP, so treat it as a first check, not proof.
What About llms.txt?
Quick note, because this comes up a lot. llms.txt is a proposed file that summarises a site for AI tools, but it doesn't replace any of the steps above. Google's AI search optimization guide says it is not needed for AI Overviews or AI Mode, and that keeping one will neither help nor harm visibility in Google Search. Some other AI tools may read it, so it is cheap to keep, but it will not fix a blocked crawler.
That's it. AI search can only cite what its crawlers are allowed to read, and a few lines in robots.txt plus one CDN setting decide that. Worth five minutes of checking.