AI crawlers don't render JavaScript – I audited my own 11 production sites

AI crawlers from OpenAI, Anthropic, and Perplexity do not execute JavaScript, leaving client-rendered single-page apps invisible to them, according to an audit by developer and product builder who runs 11 production sites. Three of the sites served AI crawlers fewer than 10 words despite perfect robots.txt files explicitly inviting them in. The developer fixed all three with build-time prerendering, boosting word counts from 5 to 1,442, 6 to 3,048, and 8 to 3,021 words.

From the Journal · July 27, 2026 Your site welcomes AI crawlers. Mine was serving them six words. Google’s crawler runs JavaScript. ChatGPT’s does not. I measured what that difference costs across eleven of my own production sites — and three of them were invisible. Key takeaways AI crawlers do not execute JavaScript. GPTBot, ClaudeBot and PerplexityBot read your raw HTML once and move on. Googlebot renders JavaScript. They do not. A client-rendered single-page app is therefore invisible to them — not badly ranked, invisible.- I measured eleven of my own production sites. Three served AI crawlers fewer than ten words. All three had perfect robots.txt files explicitly inviting those crawlers in. - I fixed all three with build-time prerendering: 5 → 1,442, 6 → 3,048, 8 → 3,021 words. The renderer took an afternoon. The guards that stop it shipping something worse took longer — and every one of them caught a real bug. - You can check any site in about four seconds. The script is below, and I would rather you ran it on your own site than took my word for anything here. The two-minute version I thought I had a content problem Last week I asked ChatGPT to recommend an app for separated parents who need court-ready records of their communication. I have built exactly that product. It has been live for months. It did not come up. I assumed I had a content problem. Not enough blog posts, not enough backlinks, the usual. I had a plumbing problem. My server was sending AI crawlers an empty room. The mechanics What is actually happening There are two kinds of crawler on the internet now, and they behave completely differently. Googlebot runs a headless Chrome. It fetches your page, executes the JavaScript, waits for the framework to build the DOM, and indexes the result. This is why single-page apps have been survivable in Google for a decade. AI crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot — do not. They issue one HTTP request, parse whatever HTML comes back, and leave. No rendering. No second attempt. No waiting. Google’s crawler runs JavaScript. ChatGPT’s does not. That single difference decides whether an AI assistant canrecommend your product. This is not speculation. An analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution. Crawler-log studies show GPTBot downloads JavaScript files in roughly 11.5% of requests and ClaudeBot in about 23.8% — and neither has ever been observed running them. They pull the file. They do not open it. So if your marketing page is a React or Vue app that boots into <div id="root" </div , here is the complete, exhaustive list of what an AI assistant knows about your company: your <title tag and your meta description. That is it. That is the whole corpus. The audit What I found on my own sites I run eight consumer products, a B2B platform, a Web3 marketplace and a studio site. I wrote a four-line script, pointed it at all eleven, and went to make coffee. I came back to this: | Site | Words served | Verdict | |---|---|---| | FamilyCare.Help | 5 | Invisible | | CoParent.Help | 6 | Invisible | | PostPilot.Help | 8 | Invisible | | Inmigrante.Help | 2,479 | Fine | | FamilyCare Facility | 2,523 | Fine | | Nexaria Digital | 2,276 | Fine | | ExamPilot.Help | 2,212 | Fine | | Zari.Help | 2,178 | Fine | | PayLess.Help | 2,116 | Fine | | Smith App Studio | 2,120 | Fine | | ProfilePhoto.Help | 1,561 | Fine | Three products. Five, six, and eight words. Now here is the part that stung. I went and looked at those three sites’ robots.txt files, expecting to find the problem there. Instead I found this: AI assistants & answer engines — explicitly welcomed for discoverability User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Google-Extended User-agent: CCBot Allow: / Immaculate. All three also serve a valid llms.txt . All three have canonical tags, Open Graph images, Twitter cards, and JSON-LD structured data in the <head . I had rolled out a red carpet, printed a welcome sign in seven languages, and pointed it all at an empty room. The eight sites that scored fine are not better marketed. They are just built on frameworks that render on the server. That is the entire difference. Two years of work on a product, and its discoverability came down to a build-time decision I made in an afternoon. Why now Why this matters more this year than last You could reasonably have ignored this in 2024. You cannot now, for three reasons that are all measurable. One: ranking and being cited have come apart. Seer Interactive analysed 5,000+ URLs across ChatGPT, Perplexity and AI Overviews and found the overlap between top Google results and AI-cited sources fell from around 70% to under 20% . Your Google rank is no longer a proxy for your AI visibility. They are now two separate distribution channels with two separate mechanics. Two: AI assistants overwhelmingly cite the brand’s own site. Yext analysed 6.8 million citations across ChatGPT, Gemini and Perplexity and found 86% came from brand-managed sources . The single most-cited source about your product is supposed to be you. If you are serving six words, you have forfeited that surface area by default. Three: this is where product discovery is moving. People no longer search “best co-parenting app” and compare ten blue links. They ask an assistant and take the shortlist. If you are not in the shortlist you are not in the consideration set, and there is no second page to be on. What llms.txt does and does not do I want to be careful here, because there is a lot of confident nonsense about this. llms.txt is a proposed convention: a plain-text file at your domain root summarising your site for language models. It is cheap, it is static, it is server-served, and I think you should have one. But it is not a fix for this problem. It is a voluntary convention with inconsistent adoption, it does not replace your actual pages in any crawler’s index, and — this is the important bit — mine were already there. All three invisible sites served a perfectly good llms.txt and it did not save them, because a 110-word summary is not a substitute for a site. Treat llms.txt as a business card. Useful. Not a building. Do this yourself Check your own site in four seconds Here is the script. It fetches a URL as GPTBot, strips scripts, styles, comments and tags, and counts what is actually left — which is what the crawler actually reads. bash /usr/bin/env bash ai-visibility.sh — how many words does an AI crawler actually see? for url in "$@"; do words=$ curl -sL --max-time 20 \ -A "Mozilla/5.0 compatible; GPTBot/1.0; +https://openai.com/gptbot " "$url" \ | perl -0777 -pe 's{<script\b. ?</script }{}gsi; s{<style\b. ?</style }{}gsi; s{<noscript\b. ?</noscript }{}gsi; s{< --. ?-- }{}gs; s{< ^ + }{ }gs' \ | tr -s " \t\r\n" " " | wc -w if "$words" -lt 200 ; then verdict="INVISIBLE" elif "$words" -lt 600 ; then verdict="THIN" else verdict="OK" fi printf "%-10s %6s words %s\n" "$verdict" "$words" "$url" done Run it: chmod +x ai-visibility.sh ./ai-visibility.sh https://yoursite.com/ https://yoursite.com/pricing Reading the result. These thresholds are judgement calls, not standards — but they have held up across everything I have tested: | Words | What it means | |---|---| Under 200 | Client-rendered. AI assistants know your title tag and nothing else. | 200–600 | Partial. Probably a hybrid app, or a page that lazy-loads its real content. | 600+ | Server-rendered. The crawler is getting your actual argument. | Two honest caveats. First, word count is a proxy for substance, not a measure of it — 2,000 words of nav links and legal boilerplate is not 2,000 words of value. Second, perl ships with macOS and Linux; on Windows use Git Bash or WSL. The fix How to fix it Ranked by effort. Start at the top and stop when your number moves. Prerender your marketing routes. For a Vite or CRA app this is usually a half-day change and it is the highest-leverage fix available. A post-build step crawls your routes in a headless browser and writes real HTML files to disk. Your app keeps working exactly as it does now; the server just has something to hand over before the JavaScript boots. Move the marketing site to a framework that renders on the server. Next.js, Astro, Remix, Nuxt. This is the durable answer if you are rebuilding anyway. Note that you only need this for the public pages — everything behind a login can stay client-rendered forever, because no crawler should be seeing it regardless. Split the marketing site from the app. Underrated. A static site on yourdomain.com and the app on app.yourdomain.com gives you perfect crawlability and a faster marketing site, and removes an entire category of routing problem. Do not reach for dynamic rendering. Serving different HTML to bots than to humans based on user-agent used to be standard advice. It is fragile, it is a cloaking risk, and prerendering solves the same problem without asking anyone to trust your user-agent detection. Then do the cheap things. Submit every sitemap to Bing Webmaster Tools — ChatGPT’s search leans on Bing’s index and this is twenty minutes of work that almost nobody does. Add Article and FAQPage schema. Put a real llms.txt up. Write your headings as questions and answer them in the first sentence, because that is the unit AI systems actually extract. Results What I did about it I built the prerenderer for all three sites. Here is the delta: | Site | Before | After | |---|---|---| | FamilyCare.Help | 5 | 1,442 | | CoParent.Help | 6 | 3,048 | | PostPilot.Help | 8 | 3,021 | Around 270 pages now ship as real HTML. Zero fall under the 200-word threshold. Three things I did not expect, all of which are the actual lessons: The naive version silently made things worse My first working run wrote 54 pages that all carried the homepage’s title, because I waited a fixed 500ms for the per-route metadata to apply and — under four concurrent headless tabs — it sometimes had not. Fifty-four URLs under one title is duplicate content. I only caught it because I diffed the titles. Prerendering created a canonical-tag bug out of nothing The HTML shell hardcodes <link rel="canonical" href="/" . Harmless when a crawler only ever fetches one file. The moment every route is its own file, every page ships two canonicals — one truthfully pointing at itself, one insisting it is the homepage. That is worse than no prerendering at all, and one of our blogs had already been deindexed by that exact bug once before. The build found a page nobody had noticed was broken PostPilot’s /platforms page had never set its own title, description or canonical — it had always served the generic homepage metadata. Every other public page had it. Nothing had ever surfaced that, because in a single-page app you cannot see the difference without going looking for it. So the honest summary is: the prerendering was the easy part. The guards were the work. If you do this, write the checks that fail your build before you write the renderer — every one of mine caught something real. Two of the first production deploys also failed outright, because the build image had no Chrome and then no system libraries to run it with. Neither failure reached anyone: the script exits non-zero on any problem, so the previous working deploy simply stayed live. That is the argument for build-time guards that fail hard rather than warn. Questions Frequently asked Do AI crawlers execute JavaScript? No. GPTBot, ClaudeBot and PerplexityBot fetch raw HTML and do not run JavaScript. Analysis of over 500 million GPTBot fetches has found no evidence of JavaScript execution. Googlebot is the exception — it renders JavaScript using a headless Chrome engine. Can ChatGPT see my React website? Only the parts present in the HTML your server sends. If your React app renders entirely in the browser, ChatGPT sees your <title and meta description and nothing else. Does robots.txt allowing GPTBot make my site visible to AI? No. robots.txt grants permission to crawl. It does not create content to crawl. A site can welcome every AI crawler in existence and still serve them an empty page. Is llms.txt enough on its own? No. It is a useful, cheap addition — a summary file for language models — but adoption is inconsistent and it does not substitute for server-rendered pages. All three of my invisible sites already had one. How do I check if my site is visible to AI crawlers? Fetch it with curl using the GPTBot user-agent and count the words in the response after stripping tags and scripts. Under 200 words on a marketing page means you are effectively invisible. The script is above. Does this hurt my Google rankings too? Less than you would think — Googlebot renders JavaScript, so a well-built single-page app can rank fine. That is exactly what makes this dangerous: your analytics look healthy while an entire emerging channel silently returns nothing. In short Summary AI crawlers read HTML. They do not run JavaScript. If your marketing site is a client-rendered single-page app, AI assistants know your title tag and nothing more — no matter how welcoming your robots.txt is. I found this on three of my own eleven production properties. The fix is a half-day of prerendering, and the first step is four seconds of curl . Run the script on your own site. If the number is under 200, you now know something about your business that your analytics were never going to tell you. Explore the apps / apps Media & support: smithappsupport@gmail.com mailto:smithappsupport@gmail.com