Why ChatGPT Can't See Your Site (Even Though Google Ranks It)

A developer's analysis reveals that AI crawlers like GPTBot do not execute JavaScript, meaning client-rendered websites appear empty to them. Testing across 500 million GPTBot fetches by Vercel and MERJ found no JavaScript execution, so content assembled in the browser is invisible to AI assistants. The post distinguishes between blocking GPTBot for training data and blocking OAI-SearchBot for search citations, warning that blanket disallows can unintentionally remove sites from AI answers.

Originally published on LetsLaunch. There is a failure mode that does not show up in any of your dashboards. Your site ranks. Analytics look normal. You open the page in a browser and it is obviously fine. And an assistant, asked about the exact thing you built, has nothing to say about you — not because it judged your product, but because it never received any words. The cause is unglamorous. Googlebot executes JavaScript. The crawlers behind ChatGPT, Claude and Perplexity do not. Vercel and MERJ measured this across more than 500 million GPTBot fetches and found no JavaScript execution at all. So a single-page app that assembles its content in the browser is two different documents depending on who asks for it: a full page for Google, and whatever sits in the initial HTML for everyone else. If that initial HTML is a <div id="root" and a script tag, that is the whole site as far as an assistant is concerned. This is worth being precise about, because "AI can't read my site" gets said in a vague way that leads people to buy the wrong fix. Nothing is being blocked, hidden or penalised. The crawler makes an ordinary HTTP request and gets an ordinary 200 response. It simply keeps what the server sent and stops there. Google's crawler does a second pass and runs the scripts; the AI crawlers have no second pass. The practical consequence is that every rendering choice you made for performance or developer convenience now has a distribution consequence you were not thinking about. Content behind a client-side fetch, a tab that loads on click, a pricing table hydrated from an API, a description rendered by a framework with no server pass — all of it is present for a human, present for Google, and absent for an assistant. Three shapes, in rough order of how often they turn up. The empty shell. The server returns 200 and about forty words: a title, a noscript line, maybe a nav. Everything that explains the product arrives later, in JavaScript. This is the classic client-rendered app, and it is the hardest to notice because nothing anywhere reports an error. The 403. Bot protection at the CDN or firewall refuses the crawler's user agent before your application is ever reached. Your robots.txt says the crawler is welcome; your edge disagrees, and the edge wins. Nobody chose this — it usually arrives as a default when someone turns on bot filtering. The accidental robots rule. Someone added a blanket disallow to keep the site out of AI training and took out the search crawler in the same line. Which brings up the distinction almost everybody gets wrong. GPTBot collects training data. OAI-SearchBot fetches pages for the index behind ChatGPT's search citations. They are independent, and blocking one has no effect on the other. Block GPTBot and you have opted out of training runs. That is a legitimate choice and it costs you nothing in ChatGPT's answers. Block OAI-SearchBot and you are removed from those answers entirely — you cannot be cited from an index you are not in. A blanket User-agent: disallow aimed at "AI" does the second thing while people believe they are doing the first. If you have ever pasted an AI-blocking robots snippet from a blog post, this is worth thirty seconds of your time. When we checked 51 startup directories https://letslaunch.today/directories for dofollow links, our checker fetched a real listing page and read the rel attribute off the outbound link. Most of what we could not verify was bot protection returning 403 — the second shape above, aimed at us. But two directories defeated the check for the first reason: they render their outbound links client-side. The anchor does not exist in the HTML that the server sends. Our dofollow checker https://letslaunch.today/free/dofollow-checker sees no link at all, and neither does anything else that does not run a browser. Those directories are almost certainly fine for humans. The point is narrower and worth sitting with: if a link only exists after JavaScript runs, then every consumer of your page that does not run JavaScript is working from a document with no link in it. The same logic that hides your product description from an assistant hides an outbound link from a crawler. You cannot see any of this in a browser, because your browser is the one thing in this story that definitely runs JavaScript. You have to fetch the page the way a crawler does. The one-liner: curl -sA "OAI-SearchBot" https://yoursite.com | wc -c curl -sA "OAI-SearchBot" https://yoursite.com | grep -c "your product description" If the byte count is small and the grep returns 0, the text is not in the document. Repeat with ClaudeBot and PerplexityBot , because a block can be per-agent, and check the status code separately — curl -o /dev/null -w "%{http code}" will tell you whether you are looking at an empty shell or a refusal. We built the AI crawler check https://letslaunch.today/free/ai-crawler-check to do this without the terminal: it fetches your page as each crawler, once, and shows the status code each one got and the text each one received. It is free and there is nothing to sign up for. If the text is missing, the fix is to make the server send it. Server-side rendering, static generation, or prerendering for bot user agents — any of the three works, and which one you pick is an ordinary engineering decision about your stack, not a marketing one. Two things that do not fix it: adding structured data it is markup about content, and it is usually injected client-side too, in which case it is equally invisible , and writing more content content the crawler cannot see does not get better by being longer . If the problem is a 403, the fix is in your CDN or firewall rules, and the tell is that robots.txt says one thing and the response code says another. Getting readable is necessary. It is not sufficient, and the gap between those two words is where most of the AI-visibility industry currently lives. Being readable means an assistant can receive your page. It does not mean an assistant will mention you. We know of no evidence that any particular action reliably causes an assistant to recommend a product, and there is a useful counterexample: G2 and Capterra carry enormous authority and get near-zero citations. Authority alone plainly does not convert into being named. Directory listings, ours included, have not been shown to cause AI citations either. What is observable is weaker and honest to state as such: consistent descriptions of a product across independent pages correlate with that product being named by assistants. Correlation is the whole of the claim. Nobody has shown causation, and anyone selling you AI visibility as a deliverable is charging for a mechanism that has not been demonstrated. It is also harder to measure than it looks. Asking a model "do you know this product" and writing down the answer is not measurement — between 9% and 28% of such decisions flip on repeat runs, even at temperature zero. A one-shot check presented as a score is a coin toss with a progress bar attached. So the sequence that survives scrutiny is short. Make sure the crawler can reach your page and receives the words on it, because that part is mechanical, verifiable and identical on every run. Then describe your product the same way everywhere it appears — your own site, your docs, and whatever independent pages list it, including a LetsLaunch listing https://letslaunch.today/submit if you want one more. Then stop, because past that point nobody can currently tell you what works. No. The crawlers behind ChatGPT, Claude and Perplexity execute no JavaScript — Vercel and MERJ confirmed this across more than 500 million GPTBot fetches. They read the HTML your server sends and nothing more. Googlebot does render, which is why a client-rendered site can rank in Google while arriving at an assistant as an empty page. No, and this is the most common confusion. GPTBot collects training data while OAI-SearchBot fetches pages for the index behind ChatGPT's search citations. They are separate crawlers, so blocking GPTBot keeps you out of training runs without affecting search. Blocking OAI-SearchBot is what removes you from ChatGPT's answers, and a blanket disallow rule does that by accident. Usually because Googlebot renders JavaScript and AI crawlers do not, so the two are reading different documents at the same URL. Fetch your page with a crawler user agent and no browser to see which one you have — an empty shell means a rendering problem, and a 403 means a bot-protection rule. Neither shows up in your analytics, because a crawler that receives nothing has nothing to report.