Why Your Website Is Invisible to ChatGPT & Perplexity — And How to Audit It in 10 Seconds A developer who audited dozens of developer-tool, SaaS and documentation sites identified four technical blockers that keep websites from being read and cited by AI search engines like ChatGPT Search, Perplexity and Claude. The blockers are robots.txt rules that block live search fetchers such as OAI-SearchBot and PerplexityBot alongside training crawlers, single-page-app routers returning HTML with a 200 status at /llms.txt, JavaScript-only pages that non-rendering AI fetchers cannot parse, and malformed or missing JSON-LD structured data. The writeup offers curl-based checks for each and recommends separating training from search user-agents, serving llms.txt as text/plain or text/markdown, and server-side rendering or pre-rendering key routes. More developers and buyers are discovering tools through ChatGPT Search, Perplexity, and Claude instead of clicking through ten blue links. If you search for advice on "Generative Engine Optimization GEO ," most articles immediately pitch vague content tips or $99/month prompt-tracking dashboards. Before worrying about prompt rankings, though, there is a much simpler question to answer: Can AI crawlers even read your website's HTML? After auditing dozens of developer tools, SaaS landing pages, and documentation sites, I kept finding the same four technical blockers. Here is how each one breaks AI visibility — and how to test your site in 10 seconds. robots.txt A lot of teams copied a blanket "block AI bots" snippet into robots.txt back in 2023 to opt out of model training. The problem is that AI platforms now separate training crawlers from live search & user-triggered fetchers . If you want to opt out of training while still appearing as a live cited source in ChatGPT Search, Perplexity, or Claude, your robots.txt needs to distinguish them: Opt out of model training crawlers optional User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / Allow live search and citation fetchers User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / Check your robots.txt right now — you might be blocking OAI-SearchBot or PerplexityBot without realizing it. /llms.txt SPA 404 Trap The llms.txt specification https://llmstxt.org/ gives LLMs a clean, Markdown index of your site at /llms.txt and optionally a full concatenated digest at /llms-full.txt . Here is a bug that catches almost every Single-Page App React, Vue, Vite, Cloudflare Pages, Netlify . Run this in your terminal against a typical SPA that doesn't have an llms.txt file yet: curl -i https://example-spa.com/llms.txt Instead of 404 Not Found , the catch-all SPA router returns: HTTP/2 200 content-type: text/html; charset=utf-8 < doctype html