# Why Your Website Is Invisible to ChatGPT & Perplexity — And How to Audit It in 10 Seconds

> Source: <https://dev.to/__bcc270a3023440da/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10-seconds-1ca8>
> Published: 2026-11-21 09:39:34+00:00

**Recommendation:** Separate training crawlers from live search fetchers in robots.txt, serve /llms.txt as text/plain or text/markdown, and pre-render or SSR key routes so non-JS fetchers see real copy.
More in the playbook "Designing agent systems": <https://wpnews.pro/playbooks/agent-architecture>

More developers and buyers are discovering tools through ChatGPT Search, Perplexity, and Claude instead of clicking through ten blue links.

If you search for advice on "Generative Engine Optimization (GEO)," most articles immediately pitch vague content tips or $99/month prompt-tracking dashboards. Before worrying about prompt rankings, though, there is a much simpler question to answer:

**Can AI crawlers even read your website's HTML?**

After auditing dozens of developer tools, SaaS landing pages, and documentation sites, I kept finding the same four technical blockers. Here is how each one breaks AI visibility — and how to test your site in 10 seconds.

`robots.txt`
A lot of teams copied a blanket "block AI bots" snippet into `robots.txt` back in 2023 to opt out of model training. The problem is that AI platforms now separate **training crawlers** from **live search & user-triggered fetchers**.

If you want to opt out of training while still appearing as a live cited source in ChatGPT Search, Perplexity, or Claude, your `robots.txt` needs to distinguish them:

```
# Opt out of model training crawlers (optional)
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Allow live search and citation fetchers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /
```

Check your `robots.txt` right now — you might be blocking `OAI-SearchBot` or `PerplexityBot` without realizing it.

`/llms.txt` SPA 404 Trap
The [llms.txt specification](https://llmstxt.org/) gives LLMs a clean, Markdown index of your site at `/llms.txt` (and optionally a full concatenated digest at `/llms-full.txt`).

Here is a bug that catches almost every Single-Page App (React, Vue, Vite, Cloudflare Pages, Netlify). Run this in your terminal against a typical SPA that doesn't have an `llms.txt` file yet:

```
curl -i https://example-spa.com/llms.txt
```

Instead of `404 Not Found`, the catch-all SPA router returns:

```
HTTP/2 200
content-type: text/html; charset=utf-8

<!doctype html>
<html><body><div id="root"></div></body></html>
```

Naive uptime or SEO checkers only check `status === 200` and report "Valid `llms.txt` detected." An AI crawler, meanwhile, receives an HTML page instead of Markdown.

A real `/llms.txt` should return `text/plain` or `text/markdown` and follow a clean heading-and-link structure:

```
# Acme API

> Acme API provides deterministic PDF invoice generation for Node.js and Python backends.

## Core Documentation
- [Quickstart](https://acme.dev/docs/quickstart): Authenticate and generate your first PDF in 3 minutes.
- [API Reference](https://acme.dev/docs/reference): Endpoints, request schemas, and webhook events.
- [Pricing](https://acme.dev/pricing): Usage tiers and rate limits.
```

Googlebot runs a massive headless Chromium rendering pipeline. Most AI search fetchers do not — they optimize for fast, cheap HTTP fetches and parse the raw HTML response directly.

Test what a non-JS fetcher actually sees on your homepage:

```
curl -s https://yoursite.com | grep -o "<p>.*</p>" | head -n 5
```

If your raw HTML is mostly `<script>` bundles and an empty `<div id="root"></div>`, AI crawlers see almost zero product copy, pricing, or documentation text. Server-side rendering (SSR), static site generation (SSG), or pre-rendering critical marketing and docs routes fixes this completely.

`JSON-LD` & Citability Signals
Structured data isn't just for Google rich snippets. Clean `JSON-LD` (`Organization`, `SoftwareApplication`, `Article`, `FAQPage`) gives extraction pipelines unambiguous facts about what your tool does, its price, its documentation URL, and who maintains it.

Common mistakes I see in the wild:

`JSON.parse()` inside `<script type="application/ld+json">`.
Because I got tired of running `curl`, checking seven `robots.txt` user-agents, inspecting raw HTML word counts, and validating `JSON-LD` by hand, I built **GEO-Audit** — a zero-dependency Node.js CLI and web auditor.

**Option A — Test in your browser (no signup):**

Paste any URL at [https://geo-audit-c8g.pages.dev](https://geo-audit-c8g.pages.dev) to get a 0–100 score across all 5 pillars plus an instant downloadable starter `llms.txt` draft.

**Option B — Run locally with the MIT CLI (Node 20+, zero `npm install`):**

```
git clone https://github.com/guyamos99/geo-audit
cd geo-audit
node cli/geo-audit.mjs https://yoursite.com
```

It checks:

`/llms-full.txt` presence + spec compliance (including SPA HTML 404 trap detection)
If you maintain a large docs site or run client audits and want to automate this across up to 500 pages locally — with auto-generated `llms.txt` + `llms-full.txt`, white-label printable HTML reports, and a GitHub Action that opens PRs when your score or docs change — there's also an optional **$29 one-time Pro toolkit** ([https://amosphere817.gumroad.com/l/geo-audit-pro](https://amosphere817.gumroad.com/l/geo-audit-pro)).

Otherwise, the free CLI and web tool cover single-page checks completely. Run your site through it and drop your score (or any weird edge cases you hit) in the comments!
