Is Your Site Even Visible to ChatGPT?

AI answer engines including ChatGPT, Claude, and Perplexity rely on distinct crawlers — GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — that many sites inadvertently block or fail to serve, according to a report tied to RankCLI, an open MCP server run via npx @rankcli/mcp-server. The report identifies five common causes of invisibility: accidental robots.txt disallow rules, client-rendered content that AI crawlers see as an empty <div id="root">, missing or broken schema.org JSON-LD, absence of an /llms.txt file, and leftover staging noindex tags. RankCLI runs GEO checks for all of these alongside a standard technical-SEO audit, with a free tier offered.

Is Your Site Even Visible to ChatGPT? You've optimized for Google. Sitemap's clean, Core Web Vitals are green, structured data validates. But when someone asks ChatGPT or Perplexity about your product, you're nowhere in the answer — and a competitor with a worse site is. The likely reason: the crawlers that feed AI answers aren't Google You've optimized for Google. Sitemap's clean, Core Web Vitals are green, structured data validates. But when someone asks ChatGPT or Perplexity about your product, you're nowhere in the answer — and a competitor with a worse site is. The likely reason: the crawlers that feed AI answers aren't Googlebot, and they don't see your site the way Googlebot does. GPTBot OpenAI — crawls for model training and, via a separate retrieval path, for real-time browsing/citation in ChatGPT answers. ClaudeBot Anthropic — same idea, powers Claude's web search and citations. PerplexityBot — crawls specifically to answer live queries with citations; this is the one most directly tied to "will I get mentioned." Google-Extended — a separate opt-in signal from regular Googlebot, controls whether Google's AI features can use your content. Each respects robots.txt, but as a distinct user-agent — blocking Googlebot doesn't block these, and blocking one doesn't block the others. Most sites have never explicitly considered any of them. 1. robots.txt blocks it, often by accident. Some site generators and "AI protection" plugins add blanket Disallow rules for GPTBot/CCBot/etc. without the owner realizing it also kills citation eligibility, not just training-data scraping. Check yours: curl -s https://yoursite.com/robots.txt | grep -A2 -i "gptbot\|claudebot\|perplexitybot" 2. The content is client-rendered only. This is the big one. Traditional SEO crawlers Googlebot execute JavaScript and wait for your SPA to render. Several AI crawlers fetch raw HTML and move on — no JS execution. If your content only exists after a React/Vue mount, an AI crawler may see an empty <div id="root" and nothing else. Test what a crawler actually receives, vs. what your browser renders: curl -s -A "GPTBot" https://yoursite.com/ | grep -o '<title . </title ' If that title tag or the body content you care about is missing or generic, but your browser shows it fine — that's your JS-rendering gap. 3. No structured data, or broken structured data. AI answer engines lean on schema.org JSON-LD to disambiguate what a page is actually about — product, article, FAQ, organization — rather than inferring it purely from prose. A page with solid content but zero structured data is harder for a retrieval system to confidently cite. 4. No llms.txt. An emerging, not-yet-universal convention — a plain-text file at /llms.txt giving a curated, LLM-readable summary of what your site is and where the important pages are, the same spirit as sitemap.xml but written for language models instead of search indexers. Not every crawler uses it yet, but it costs almost nothing to add and directly addresses the "can a model even summarize what we do" problem. 5. Stale noindex leftovers. Staging-environment meta tags <meta name="robots" content="noindex" that never got removed after launch block everything — traditional and AI crawlers alike. curl -A "GPTBot" yoursite.com/robots.txt — confirm you're not accidentally blocked. curl -A "GPTBot" yoursite.com/ and diff it against what your browser renders — confirm the content is actually in the raw HTML. Validate your structured data Google's Rich Results Test works fine for this even though it's Google-branded — it's just a schema.org validator . Check if /llms.txt exists. If not, it's a 20-minute add. Grep your codebase for leftover noindex tags. We ran into this exact problem building RankCLI — the open MCP server npx @rankcli/mcp-server, no signup runs GEO checks for all of the above alongside the usual technical-SEO audit, so it's one pass instead of five manual curl commands every time you ship. Free tier is genuinely free; the hosted layer just adds scheduling and auto-fix PRs on top. But even without any tool, the five checks above take less time than writing this sentence took, and most sites have never run them once. Key Takeaways - •You've optimized for Google - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/rankcli/is-your-site-even-visible-to-chatgpt-281f