{"slug": "why-chatgpt-can-t-see-your-site-even-though-google-ranks-it", "title": "Why ChatGPT Can't See Your Site (Even Though Google Ranks It)", "summary": "A developer's analysis reveals that AI crawlers like GPTBot do not execute JavaScript, meaning client-rendered websites appear empty to them. Testing across 500 million GPTBot fetches by Vercel and MERJ found no JavaScript execution, so content assembled in the browser is invisible to AI assistants. The post distinguishes between blocking GPTBot for training data and blocking OAI-SearchBot for search citations, warning that blanket disallows can unintentionally remove sites from AI answers.", "body_md": "*Originally published on LetsLaunch.*\n\nThere is a failure mode that does not show up in any of your dashboards. Your\n\nsite ranks. Analytics look normal. You open the page in a browser and it is\n\nobviously fine. And an assistant, asked about the exact thing you built, has\n\nnothing to say about you — not because it judged your product, but because it\n\nnever received any words.\n\nThe cause is unglamorous. Googlebot executes JavaScript. The crawlers behind\n\nChatGPT, Claude and Perplexity do not. Vercel and MERJ measured this across\n\nmore than 500 million GPTBot fetches and found no JavaScript execution at all.\n\nSo a single-page app that assembles its content in the browser is two different\n\ndocuments depending on who asks for it: a full page for Google, and whatever\n\nsits in the initial HTML for everyone else.\n\nIf that initial HTML is a `<div id=\"root\">`\n\nand a script tag, that is the whole\n\nsite as far as an assistant is concerned.\n\nThis is worth being precise about, because \"AI can't read my site\" gets said in\n\na vague way that leads people to buy the wrong fix.\n\nNothing is being blocked, hidden or penalised. The crawler makes an ordinary\n\nHTTP request and gets an ordinary 200 response. It simply keeps what the server\n\nsent and stops there. Google's crawler does a second pass and runs the scripts;\n\nthe AI crawlers have no second pass.\n\nThe practical consequence is that every rendering choice you made for\n\nperformance or developer convenience now has a distribution consequence you\n\nwere not thinking about. Content behind a client-side fetch, a tab that loads\n\non click, a pricing table hydrated from an API, a description rendered by a\n\nframework with no server pass — all of it is present for a human, present for\n\nGoogle, and absent for an assistant.\n\nThree shapes, in rough order of how often they turn up.\n\n**The empty shell.** The server returns 200 and about forty words: a title, a\n\nnoscript line, maybe a nav. Everything that explains the product arrives later,\n\nin JavaScript. This is the classic client-rendered app, and it is the hardest\n\nto notice because nothing anywhere reports an error.\n\n**The 403.** Bot protection at the CDN or firewall refuses the crawler's user\n\nagent before your application is ever reached. Your `robots.txt`\n\nsays the\n\ncrawler is welcome; your edge disagrees, and the edge wins. Nobody chose this —\n\nit usually arrives as a default when someone turns on bot filtering.\n\n**The accidental robots rule.** Someone added a blanket disallow to keep the\n\nsite out of AI training and took out the search crawler in the same line. Which\n\nbrings up the distinction almost everybody gets wrong.\n\n`GPTBot`\n\ncollects training data. `OAI-SearchBot`\n\nfetches pages for the index\n\nbehind ChatGPT's search citations. They are independent, and blocking one has\n\nno effect on the other.\n\nBlock `GPTBot`\n\nand you have opted out of training runs. That is a legitimate\n\nchoice and it costs you nothing in ChatGPT's answers. Block `OAI-SearchBot`\n\nand\n\nyou are removed from those answers entirely — you cannot be cited from an index\n\nyou are not in.\n\nA blanket `User-agent: *`\n\ndisallow aimed at \"AI\" does the second thing while\n\npeople believe they are doing the first. If you have ever pasted an AI-blocking\n\nrobots snippet from a blog post, this is worth thirty seconds of your time.\n\nWhen we checked [51 startup directories](https://letslaunch.today/directories) for dofollow links, our\n\nchecker fetched a real listing page and read the `rel`\n\nattribute off the\n\noutbound link. Most of what we could not verify was bot protection returning\n\n403 — the second shape above, aimed at us.\n\nBut two directories defeated the check for the first reason: they render their\n\noutbound links client-side. The anchor does not exist in the HTML that the\n\nserver sends. Our [dofollow checker](https://letslaunch.today/free/dofollow-checker) sees no link at\n\nall, and neither does anything else that does not run a browser.\n\nThose directories are almost certainly fine for humans. The point is narrower\n\nand worth sitting with: if a link only exists after JavaScript runs, then every\n\nconsumer of your page that does not run JavaScript is working from a document\n\nwith no link in it. The same logic that hides your product description from an\n\nassistant hides an outbound link from a crawler.\n\nYou cannot see any of this in a browser, because your browser is the one thing\n\nin this story that definitely runs JavaScript. You have to fetch the page the\n\nway a crawler does.\n\nThe one-liner:\n\n```\ncurl -sA \"OAI-SearchBot\" https://yoursite.com | wc -c\ncurl -sA \"OAI-SearchBot\" https://yoursite.com | grep -c \"your product description\"\n```\n\nIf the byte count is small and the grep returns 0, the text is not in the\n\ndocument. Repeat with `ClaudeBot`\n\nand `PerplexityBot`\n\n, because a block can be\n\nper-agent, and check the status code separately — `curl -o /dev/null -w \"%{http_code}\"`\n\nwill tell you whether you are looking at an empty shell or a refusal.\n\nWe built [the AI crawler check](https://letslaunch.today/free/ai-crawler-check) to do this without the\n\nterminal: it fetches your page as each crawler, once, and shows the status code\n\neach one got and the text each one received. It is free and there is nothing to\n\nsign up for.\n\nIf the text is missing, the fix is to make the server send it. Server-side\n\nrendering, static generation, or prerendering for bot user agents — any of the\n\nthree works, and which one you pick is an ordinary engineering decision about\n\nyour stack, not a marketing one.\n\nTwo things that do not fix it: adding structured data (it is markup about\n\ncontent, and it is usually injected client-side too, in which case it is\n\nequally invisible), and writing more content (content the crawler cannot see\n\ndoes not get better by being longer).\n\nIf the problem is a 403, the fix is in your CDN or firewall rules, and the\n\ntell is that `robots.txt`\n\nsays one thing and the response code says another.\n\nGetting readable is necessary. It is not sufficient, and the gap between those\n\ntwo words is where most of the AI-visibility industry currently lives.\n\nBeing readable means an assistant *can* receive your page. It does not mean an\n\nassistant will mention you. We know of no evidence that any particular action\n\nreliably causes an assistant to recommend a product, and there is a useful\n\ncounterexample: G2 and Capterra carry enormous authority and get near-zero\n\ncitations. Authority alone plainly does not convert into being named.\n\nDirectory listings, ours included, have not been shown to cause AI citations\n\neither. What is observable is weaker and honest to state as such: consistent\n\ndescriptions of a product across independent pages correlate with that product\n\nbeing named by assistants. Correlation is the whole of the claim. Nobody has\n\nshown causation, and anyone selling you AI visibility as a deliverable is\n\ncharging for a mechanism that has not been demonstrated.\n\nIt is also harder to measure than it looks. Asking a model \"do you know this\n\nproduct\" and writing down the answer is not measurement — between 9% and 28% of\n\nsuch decisions flip on repeat runs, even at temperature zero. A one-shot check\n\npresented as a score is a coin toss with a progress bar attached.\n\nSo the sequence that survives scrutiny is short. Make sure the crawler can\n\nreach your page and receives the words on it, because that part is mechanical,\n\nverifiable and identical on every run. Then describe your product the same way\n\neverywhere it appears — your own site, your docs, and whatever independent\n\npages list it, including a [LetsLaunch listing](https://letslaunch.today/submit) if you want one more.\n\nThen stop, because past that point nobody can currently tell you what works.\n\nNo. The crawlers behind ChatGPT, Claude and Perplexity execute no JavaScript —\n\nVercel and MERJ confirmed this across more than 500 million GPTBot fetches.\n\nThey read the HTML your server sends and nothing more. Googlebot does render,\n\nwhich is why a client-rendered site can rank in Google while arriving at an\n\nassistant as an empty page.\n\nNo, and this is the most common confusion. `GPTBot`\n\ncollects training data\n\nwhile `OAI-SearchBot`\n\nfetches pages for the index behind ChatGPT's search\n\ncitations. They are separate crawlers, so blocking `GPTBot`\n\nkeeps you out of\n\ntraining runs without affecting search. Blocking `OAI-SearchBot`\n\nis what\n\nremoves you from ChatGPT's answers, and a blanket disallow rule does that by\n\naccident.\n\nUsually because Googlebot renders JavaScript and AI crawlers do not, so the two\n\nare reading different documents at the same URL. Fetch your page with a crawler\n\nuser agent and no browser to see which one you have — an empty shell means a\n\nrendering problem, and a 403 means a bot-protection rule. Neither shows up in\n\nyour analytics, because a crawler that receives nothing has nothing to report.", "url": "https://wpnews.pro/news/why-chatgpt-can-t-see-your-site-even-though-google-ranks-it", "canonical_source": "https://dev.to/costin_gheorghe_40d2a06d8/why-chatgpt-cant-see-your-site-even-though-google-ranks-it-3gcn", "published_at": "2026-09-03 12:22:11+00:00", "updated_at": "2026-09-03 12:55:23.711385+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["ChatGPT", "Claude", "Perplexity", "Googlebot", "GPTBot", "OAI-SearchBot", "Vercel", "MERJ"], "alternates": {"html": "https://wpnews.pro/news/why-chatgpt-can-t-see-your-site-even-though-google-ranks-it", "markdown": "https://wpnews.pro/news/why-chatgpt-can-t-see-your-site-even-though-google-ranks-it.md", "text": "https://wpnews.pro/news/why-chatgpt-can-t-see-your-site-even-though-google-ranks-it.txt", "jsonld": "https://wpnews.pro/news/why-chatgpt-can-t-see-your-site-even-though-google-ranks-it.jsonld"}}