Can ChatGPT, Claude and Perplexity actually crawl your site? A 10-minute check

A developer built BotReady, a set of free checks for AI crawler access, and published a manual version of the checklist for verifying whether ChatGPT, Claude and Perplexity bots can reach a site. The guide distinguishes training crawlers like GPTBot and ClaudeBot from search and user-triggered fetchers, and walks through robots.txt rules, CDN/WAF blocks that return 403, JavaScript-rendered pages that serve empty HTML to crawlers, and access-log verification.

More and more people find websites through AI assistants instead of a classic list of blue links. If an assistant can't fetch your pages, it can't quote them, summarize them or link to them. The annoying part: you can be blocked without ever deciding to block anyone. I ran into this while building BotReady https://botready.app/ , a set of free checks for AI crawler access. Below is the manual version of the same checklist, so you can verify your own site with nothing more than a browser and a terminal. AI companies usually run several user agents, and they do different jobs. Blocking one does not block the others. | Company | Training crawler | Search / indexing | User-triggered fetch | |---|---|---|---| | OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | | Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | | Perplexity | — | PerplexityBot | Perplexity-User | | | Google-Extended robots.txt token, not a separate bot | Googlebot | — | | Apple | Applebot-Extended token | Applebot | — | | Common Crawl | CCBot | — | — | The practical takeaway: "I don't want my content used for training" and "I want to show up when someone asks ChatGPT about my product" are two different robots.txt decisions . Many sites block everything with one rule and lose visibility they actually wanted. Also note that user-triggered fetchers a person pasting your URL into a chat are treated differently by some vendors and may not follow robots.txt the same way a crawler does. Check each vendor's docs for the current behaviour. Open https://yourdomain.com/robots.txt and look for three things: User-agent: + Disallow: / blocks everyone, including AI bots . Disallow rules for the bots above, often added by a CMS plugin or a security preset you forgot about. User-agent group that matches it and ignores the rest, so a GPTBot group without an Allow can override your generous group. A common "search yes, training no" setup looks like this: Allow AI search / answer engines User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / Opt out of model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / User-agent: Allow: / Sitemap: https://yourdomain.com/sitemap.xml This is a policy choice, not a recommendation: decide what you want, then make the file say exactly that. This is the one that surprises people most. robots.txt is a polite request. A CDN or WAF rule is a hard block that returns 403 before your server is even involved. A quick first test from your terminal: curl -s -o /dev/null -w "%{http code}\n" \ -A "Mozilla/5.0 compatible; GPTBot/1.2; +https://openai.com/gptbot " \ https://yourdomain.com/ Repeat with ClaudeBot , PerplexityBot , etc. 200 is good; 403 , 429 or 503 deserves a look. One caveat: spoofing the user agent from your laptop is not the same as a real crawler request. Many firewalls verify bots by IP range, so a fake GPTBot may be blocked while the real one is allowed, or the other way around. The test tells you whether UA-based rules exist; your logs tell you the truth step 5 . Most AI crawlers fetch raw HTML and do not run your JavaScript. If your page is a client-side SPA, they may get an empty <div id="root" </div . curl -s https://yourdomain.com/your-page | grep -i "a sentence from your page" If the text you care about isn't in the raw HTML, consider SSR, static generation or prerendering for content pages. While you're there, check that the page has a clear <title , one <h1 , a meta description and valid structured data. These help any machine reader, not only AI ones. This is the only place you see what really happened. Search access logs for the bot names: grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user" access.log \ | awk '{print $9}' | sort | uniq -c That prints the status codes each bot received. Lots of 403 s mean something is blocking them. No hits at all may mean they never got past the CDN check CDN logs or simply haven't discovered the site yet. Anyone can fake a user agent, so verify important hits against the vendor's published IP ranges. /llms.txt is a proposed Markdown file that gives AI tools a short map of your most important pages. It's not a standard that crawlers are required to use, and it doesn't replace robots.txt or a sitemap. It's cheap to add, though, and handy for documentation sites. If you'd rather not do it by hand, the AI Crawler Checker https://botready.app/ai-crawler-checker/ and AI Visibility Checker https://botready.app/ai-visibility-checker/ on BotReady run these checks for a URL in one go, and the robots.txt generator https://botready.app/ai-robots-txt-generator/ writes the per-bot rules for you. Free, no sign-up for the first checks. What did you find on your own site? I'm curious how many people discover a forgotten firewall rule. 👀