# Can ChatGPT, Claude and Perplexity actually crawl your site? A 10-minute check > Source: > Published: 2026-11-13 01:12:13+00:00 More and more people find websites through AI assistants instead of a classic list of blue links. If an assistant can't fetch your pages, it can't quote them, summarize them or link to them. The annoying part: you can be blocked without ever deciding to block anyone. I ran into this while building [BotReady](https://botready.app/), a set of free checks for AI crawler access. Below is the manual version of the same checklist, so you can verify your own site with nothing more than a browser and a terminal. AI companies usually run **several** user agents, and they do different jobs. Blocking one does not block the others. | Company | Training crawler | Search / indexing | User-triggered fetch | |---|---|---|---| | OpenAI | `GPTBot` | `OAI-SearchBot` | `ChatGPT-User` | | Anthropic | `ClaudeBot` | `Claude-SearchBot` | `Claude-User` | | Perplexity | — | `PerplexityBot` | `Perplexity-User` | | | `Google-Extended` (robots.txt token, not a separate bot) | `Googlebot` | — | | Apple | `Applebot-Extended` (token) | `Applebot` | — | | Common Crawl | `CCBot` | — | — | The practical takeaway: "I don't want my content used for training" and "I want to show up when someone asks ChatGPT about my product" are **two different robots.txt decisions**. Many sites block everything with one rule and lose visibility they actually wanted. Also note that user-triggered fetchers (a person pasting your URL into a chat) are treated differently by some vendors and may not follow robots.txt the same way a crawler does. Check each vendor's docs for the current behaviour. Open `https://yourdomain.com/robots.txt` and look for three things: `User-agent: *` + `Disallow: /` (blocks everyone, including AI bots).`Disallow` rules for the bots above, often added by a CMS plugin or a security preset you forgot about.`User-agent` group that matches it and ignores the rest, so a `GPTBot` group without an `Allow` can override your generous `*` group. A common "search yes, training no" setup looks like this: ``` # Allow AI search / answer engines User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / # Opt out of model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / User-agent: * Allow: / Sitemap: https://yourdomain.com/sitemap.xml ``` This is a policy choice, not a recommendation: decide what you want, then make the file say exactly that. This is the one that surprises people most. robots.txt is a polite request. A CDN or WAF rule is a hard block that returns `403` before your server is even involved. A quick first test from your terminal: ``` curl -s -o /dev/null -w "%{http_code}\n" \ -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \ https://yourdomain.com/ ``` Repeat with `ClaudeBot`, `PerplexityBot`, etc. `200` is good; `403`, `429` or `503` deserves a look. One caveat: spoofing the user agent from your laptop is **not** the same as a real crawler request. Many firewalls verify bots by IP range, so a fake GPTBot may be blocked while the real one is allowed, or the other way around. The test tells you whether UA-based rules exist; your logs tell you the truth (step 5). Most AI crawlers fetch raw HTML and do not run your JavaScript. If your page is a client-side SPA, they may get an empty `
`. ``` curl -s https://yourdomain.com/your-page | grep -i "a sentence from your page" ``` If the text you care about isn't in the raw HTML, consider SSR, static generation or prerendering for content pages. While you're there, check that the page has a clear ``, one `<h1>`, a meta description and valid structured data. These help any machine reader, not only AI ones. This is the only place you see what really happened. Search access logs for the bot names: ``` grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user" access.log \ | awk '{print $9}' | sort | uniq -c ``` That prints the status codes each bot received. Lots of `403` s mean something is blocking them. No hits at all may mean they never got past the CDN (check CDN logs) or simply haven't discovered the site yet. Anyone can fake a user agent, so verify important hits against the vendor's published IP ranges. `/llms.txt` is a proposed Markdown file that gives AI tools a short map of your most important pages. It's not a standard that crawlers are required to use, and it doesn't replace robots.txt or a sitemap. It's cheap to add, though, and handy for documentation sites. If you'd rather not do it by hand, the [AI Crawler Checker](https://botready.app/ai-crawler-checker/) and [AI Visibility Checker](https://botready.app/ai-visibility-checker/) on BotReady run these checks for a URL in one go, and the [robots.txt generator](https://botready.app/ai-robots-txt-generator/) writes the per-bot rules for you. Free, no sign-up for the first checks. What did you find on your own site? I'm curious how many people discover a forgotten firewall rule. 👀