cd /news/generative-engine-optimization/why-your-website-is-invisible-to-cha… · home › topics › generative-engine-optimization › article
[ARTICLE · art-173718] src=dev.to ↗ pub= topic=generative-engine-optimization verified=true sentiment=· neutral

Why Your Website Is Invisible to ChatGPT & Perplexity — And How to Audit It in 10 Seconds

A developer who audited dozens of developer-tool, SaaS and documentation sites identified four technical blockers that keep websites from being read and cited by AI search engines like ChatGPT Search, Perplexity and Claude. The blockers are robots.txt rules that block live search fetchers such as OAI-SearchBot and PerplexityBot alongside training crawlers, single-page-app routers returning HTML with a 200 status at /llms.txt, JavaScript-only pages that non-rendering AI fetchers cannot parse, and malformed or missing JSON-LD structured data. The writeup offers curl-based checks for each and recommends separating training from search user-agents, serving llms.txt as text/plain or text/markdown, and server-side rendering or pre-rendering key routes.

recommendation

Separate training crawlers from live search fetchers in robots.txt, serve /llms.txt as text/plain or text/markdown, and pre-render or SSR key routes so non-JS fetchers see real copy.

more in playbook: Designing agent systems
read4 min views7 publishedNov 21, 2026

More developers and buyers are discovering tools through ChatGPT Search, Perplexity, and Claude instead of clicking through ten blue links.

If you search for advice on "Generative Engine Optimization (GEO)," most articles immediately pitch vague content tips or $99/month prompt-tracking dashboards. Before worrying about prompt rankings, though, there is a much simpler question to answer:

Can AI crawlers even read your website's HTML?

After auditing dozens of developer tools, SaaS landing pages, and documentation sites, I kept finding the same four technical blockers. Here is how each one breaks AI visibility — and how to test your site in 10 seconds.

robots.txt A lot of teams copied a blanket "block AI bots" snippet into robots.txt back in 2023 to opt out of model training. The problem is that AI platforms now separate training crawlers from live search & user-triggered fetchers.

If you want to opt out of training while still appearing as a live cited source in ChatGPT Search, Perplexity, or Claude, your robots.txt needs to distinguish them:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

Check your robots.txt right now — you might be blocking OAI-SearchBot or PerplexityBot without realizing it.

/llms.txt SPA 404 Trap The llms.txt specification gives LLMs a clean, Markdown index of your site at /llms.txt (and optionally a full concatenated digest at /llms-full.txt).

Here is a bug that catches almost every Single-Page App (React, Vue, Vite, Cloudflare Pages, Netlify). Run this in your terminal against a typical SPA that doesn't have an llms.txt file yet:

curl -i https://example-spa.com/llms.txt

Instead of 404 Not Found, the catch-all SPA router returns:

HTTP/2 200
content-type: text/html; charset=utf-8

<!doctype html>
<html><body><div id="root"></div></body></html>

Naive uptime or SEO checkers only check status === 200 and report "Valid llms.txt detected." An AI crawler, meanwhile, receives an HTML page instead of Markdown.

A real /llms.txt should return text/plain or text/markdown and follow a clean heading-and-link structure:


> Acme API provides deterministic PDF invoice generation for Node.js and Python backends.

## Core Documentation
- [Quickstart](https://acme.dev/docs/quickstart): Authenticate and generate your first PDF in 3 minutes.
- [API Reference](https://acme.dev/docs/reference): Endpoints, request schemas, and webhook events.
- [Pricing](https://acme.dev/pricing): Usage tiers and rate limits.

Googlebot runs a massive headless Chromium rendering pipeline. Most AI search fetchers do not — they optimize for fast, cheap HTTP fetches and parse the raw HTML response directly.

Test what a non-JS fetcher actually sees on your homepage:

curl -s https://yoursite.com | grep -o "<p>.*</p>" | head -n 5

If your raw HTML is mostly <script> bundles and an empty <div id="root"></div>, AI crawlers see almost zero product copy, pricing, or documentation text. Server-side rendering (SSR), static site generation (SSG), or pre-rendering critical marketing and docs routes fixes this completely.

JSON-LD & Citability Signals Structured data isn't just for Google rich snippets. Clean JSON-LD (Organization, SoftwareApplication, Article, FAQPage) gives extraction pipelines unambiguous facts about what your tool does, its price, its documentation URL, and who maintains it.

Common mistakes I see in the wild:

JSON.parse() inside <script type="application/ld+json">. Because I got tired of running curl, checking seven robots.txt user-agents, inspecting raw HTML word counts, and validating JSON-LD by hand, I built GEO-Audit — a zero-dependency Node.js CLI and web auditor.

Option A — Test in your browser (no signup):

Paste any URL at https://geo-audit-c8g.pages.dev to get a 0–100 score across all 5 pillars plus an instant downloadable starter llms.txt draft.

Option B — Run locally with the MIT CLI (Node 20+, zero npm install):

git clone https://github.com/guyamos99/geo-audit
cd geo-audit
node cli/geo-audit.mjs https://yoursite.com

It checks:

/llms-full.txt presence + spec compliance (including SPA HTML 404 trap detection) If you maintain a large docs site or run client audits and want to automate this across up to 500 pages locally — with auto-generated llms.txt + llms-full.txt, white-label printable HTML reports, and a GitHub Action that opens PRs when your score or docs change — there's also an optional $29 one-time Pro toolkit (https://amosphere817.gumroad.com/l/geo-audit-pro).

Otherwise, the free CLI and web tool cover single-page checks completely. Run your site through it and drop your score (or any weird edge cases you hit) in the comments!

── more in #generative-engine-optimization 4 stories · sorted by recency
── more on @chatgpt search 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/why-your-website-is-…] indexed:0 read:4min 2026-11-21 · —