{"slug": "why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10", "title": "Why Your Website Is Invisible to ChatGPT & Perplexity — And How to Audit It in 10 Seconds", "summary": "A developer who audited dozens of developer-tool, SaaS and documentation sites identified four technical blockers that keep websites from being read and cited by AI search engines like ChatGPT Search, Perplexity and Claude. The blockers are robots.txt rules that block live search fetchers such as OAI-SearchBot and PerplexityBot alongside training crawlers, single-page-app routers returning HTML with a 200 status at /llms.txt, JavaScript-only pages that non-rendering AI fetchers cannot parse, and malformed or missing JSON-LD structured data. The writeup offers curl-based checks for each and recommends separating training from search user-agents, serving llms.txt as text/plain or text/markdown, and server-side rendering or pre-rendering key routes.", "body_md": "More developers and buyers are discovering tools through ChatGPT Search, Perplexity, and Claude instead of clicking through ten blue links.\n\nIf you search for advice on \"Generative Engine Optimization (GEO),\" most articles immediately pitch vague content tips or $99/month prompt-tracking dashboards. Before worrying about prompt rankings, though, there is a much simpler question to answer:\n\n**Can AI crawlers even read your website's HTML?**\n\nAfter auditing dozens of developer tools, SaaS landing pages, and documentation sites, I kept finding the same four technical blockers. Here is how each one breaks AI visibility — and how to test your site in 10 seconds.\n\n`robots.txt`\nA lot of teams copied a blanket \"block AI bots\" snippet into `robots.txt` back in 2023 to opt out of model training. The problem is that AI platforms now separate **training crawlers** from **live search & user-triggered fetchers**.\n\nIf you want to opt out of training while still appearing as a live cited source in ChatGPT Search, Perplexity, or Claude, your `robots.txt` needs to distinguish them:\n\n```\n# Opt out of model training crawlers (optional)\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\n# Allow live search and citation fetchers\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n```\n\nCheck your `robots.txt` right now — you might be blocking `OAI-SearchBot` or `PerplexityBot` without realizing it.\n\n`/llms.txt` SPA 404 Trap\nThe [llms.txt specification](https://llmstxt.org/) gives LLMs a clean, Markdown index of your site at `/llms.txt` (and optionally a full concatenated digest at `/llms-full.txt`).\n\nHere is a bug that catches almost every Single-Page App (React, Vue, Vite, Cloudflare Pages, Netlify). Run this in your terminal against a typical SPA that doesn't have an `llms.txt` file yet:\n\n```\ncurl -i https://example-spa.com/llms.txt\n```\n\nInstead of `404 Not Found`, the catch-all SPA router returns:\n\n```\nHTTP/2 200\ncontent-type: text/html; charset=utf-8\n\n<!doctype html>\n<html><body><div id=\"root\"></div></body></html>\n```\n\nNaive uptime or SEO checkers only check `status === 200` and report \"Valid `llms.txt` detected.\" An AI crawler, meanwhile, receives an HTML page instead of Markdown.\n\nA real `/llms.txt` should return `text/plain` or `text/markdown` and follow a clean heading-and-link structure:\n\n```\n# Acme API\n\n> Acme API provides deterministic PDF invoice generation for Node.js and Python backends.\n\n## Core Documentation\n- [Quickstart](https://acme.dev/docs/quickstart): Authenticate and generate your first PDF in 3 minutes.\n- [API Reference](https://acme.dev/docs/reference): Endpoints, request schemas, and webhook events.\n- [Pricing](https://acme.dev/pricing): Usage tiers and rate limits.\n```\n\nGooglebot runs a massive headless Chromium rendering pipeline. Most AI search fetchers do not — they optimize for fast, cheap HTTP fetches and parse the raw HTML response directly.\n\nTest what a non-JS fetcher actually sees on your homepage:\n\n```\ncurl -s https://yoursite.com | grep -o \"<p>.*</p>\" | head -n 5\n```\n\nIf your raw HTML is mostly `<script>` bundles and an empty `<div id=\"root\"></div>`, AI crawlers see almost zero product copy, pricing, or documentation text. Server-side rendering (SSR), static site generation (SSG), or pre-rendering critical marketing and docs routes fixes this completely.\n\n`JSON-LD` & Citability Signals\nStructured data isn't just for Google rich snippets. Clean `JSON-LD` (`Organization`, `SoftwareApplication`, `Article`, `FAQPage`) gives extraction pipelines unambiguous facts about what your tool does, its price, its documentation URL, and who maintains it.\n\nCommon mistakes I see in the wild:\n\n`JSON.parse()` inside `<script type=\"application/ld+json\">`.\nBecause I got tired of running `curl`, checking seven `robots.txt` user-agents, inspecting raw HTML word counts, and validating `JSON-LD` by hand, I built **GEO-Audit** — a zero-dependency Node.js CLI and web auditor.\n\n**Option A — Test in your browser (no signup):**\n\nPaste any URL at [https://geo-audit-c8g.pages.dev](https://geo-audit-c8g.pages.dev) to get a 0–100 score across all 5 pillars plus an instant downloadable starter `llms.txt` draft.\n\n**Option B — Run locally with the MIT CLI (Node 20+, zero `npm install`):**\n\n```\ngit clone https://github.com/guyamos99/geo-audit\ncd geo-audit\nnode cli/geo-audit.mjs https://yoursite.com\n```\n\nIt checks:\n\n`/llms-full.txt` presence + spec compliance (including SPA HTML 404 trap detection)\nIf you maintain a large docs site or run client audits and want to automate this across up to 500 pages locally — with auto-generated `llms.txt` + `llms-full.txt`, white-label printable HTML reports, and a GitHub Action that opens PRs when your score or docs change — there's also an optional **$29 one-time Pro toolkit** ([https://amosphere817.gumroad.com/l/geo-audit-pro](https://amosphere817.gumroad.com/l/geo-audit-pro)).\n\nOtherwise, the free CLI and web tool cover single-page checks completely. Run your site through it and drop your score (or any weird edge cases you hit) in the comments!", "url": "https://wpnews.pro/news/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10", "canonical_source": "https://dev.to/__bcc270a3023440da/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10-seconds-1ca8", "published_at": "2026-11-21 09:39:34+00:00", "updated_at": "2026-11-21 09:42:06.782561+00:00", "lang": "en", "topics": ["generative-engine-optimization", "ai-crawlers", "ai-search", "agent-protocols", "structured-data"], "entities": ["ChatGPT Search", "Perplexity", "Claude", "GPTBot", "OAI-SearchBot", "PerplexityBot", "Googlebot"], "also_reported_by": [], "takeaway": "AI search fetchers parse raw HTML without JavaScript rendering, so sites that return an empty SPA root div or block OAI-SearchBot and PerplexityBot in robots.txt are invisible to citation pipelines even when Googlebot renders them fine.", "recommendation": {"advice": "Separate training crawlers from live search fetchers in robots.txt, serve /llms.txt as text/plain or text/markdown, and pre-render or SSR key routes so non-JS fetchers see real copy.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10", "markdown": "https://wpnews.pro/news/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10.md", "text": "https://wpnews.pro/news/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10.txt", "jsonld": "https://wpnews.pro/news/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10.jsonld"}}