Run this against your site right now:
curl -s https://yoursite.com | grep -o "<h1[^>]*>.*</h1>"
If nothing comes back, or you get an empty <div id="root">
, then large parts of the internet cannot read your site. Not "reads it poorly". Cannot read it.
I do this on every site we take over, and the result surprises people often enough that it is worth writing down.
curl
does exactly one thing: it fetches HTML and stops. It does not run JavaScript. It does not wait for hydration. It does not call your API.
That is also what a large number of crawlers do.
Googlebot is the exception people think of, and it is genuinely good: it fetches, queues the page, and renders it with a headless browser later. Client rendered content usually gets indexed eventually.
The AI crawlers are a different story. As of now, the major ones (GPTBot, ClaudeBot, PerplexityBot, and friends) largely do not execute JavaScript. They fetch the HTML, take what is in it, and move on. Whatever your framework paints after the bundle loads is invisible to them.
So curl
is a decent proxy for the floor: if your content is not in that response, assume a meaningful slice of automated readers never see it.
For years the bet was reasonable. Google renders JS, Google is search, so client rendering was survivable.
Then a chunk of discovery moved to assistants. People ask ChatGPT for a recommendation instead of scrolling ten blue links. If the model cannot read your page, you are not in the answer, and there is no page two to be on.
For a marketing site this is the whole ballgame. For a small business it is worse, because the queries that matter ("web designers in X", "who does Y near me") are precisely the ones people now ask an assistant.
1. Raw HTML, by word count.
curl -s https://yoursite.com | wc -c # total bytes
curl -s https://yoursite.com | \
sed 's/<script[^>]*>.*<\/script>//g' | \
sed 's/<[^>]*>/ /g' | wc -w # actual visible words
A marketing homepage with 40 words of real text in the raw HTML is a red flag. A React SPA often returns fewer than 10.
2. Compare raw against rendered.
// in DevTools console, on your live site
document.body.innerText.split(/\s+/).length
Compare that number to the wc -w
above. A large gap is the amount of your site that only exists after JavaScript runs.
3. Ask with the right user agent.
curl -s -A "GPTBot" https://yoursite.com | grep -c "your-key-phrase"
Also worth confirming you are not blocking these crawlers by accident. Plenty of robots.txt
files quietly disallow them because someone pasted a blocklist from a blog post in 2023:
User-agent: GPTBot
Disallow: /
That is a choice you can make deliberately. Making it by accident is expensive.
The fix is not "switch frameworks". It is "make sure the words are in the first response".
Next.js App Router. Server Components are the default, so you are mostly fine until someone adds "use client"
at the top of a page to use one hook. Push "use client"
down to the leaf that needs it instead of the page.
// page.tsx stays a server component
export default async function Page() {
const data = await getData();
return (
<>
<h1>{data.title}</h1>
<p>{data.summary}</p>
<InteractiveWidget /> {/* only this is "use client" */}
</>
);
}
The heading and copy are now in the HTML. The widget hydrates after. Both readers are served.
Vite, CRA, plain SPA. You need prerendering. vite-plugin-ssr
/ Vike, or a prerender step for known routes, or move the marketing pages to static HTML and keep the SPA for the app itself. The split is usually the honest answer: your dashboard does not need to be crawlable, your pricing page does.
Astro / Eleventy / Hugo. Already static. Check anyway, because a client only island can still swallow your main content if someone got enthusiastic.
Anything behind a cookie banner that blanks the page. Some consent implementations render an interstitial and nothing else until a click. Crawlers do not click.
Getting the HTML right is necessary but it is not sufficient. Assistants quote text. Give them text worth quoting.
<h1>
that states what you do, in the words a customer would use, not a slogan.We rebuilt a manufacturer's site last year where the entire product range lived inside a JavaScript carousel. The raw HTML contained the company name and a cookie notice. Moving the product copy into server rendered markup was most of the work, and it was not a redesign, it was a text problem wearing a framework costume.
curl -s https://yoursite.com | sed 's/<[^>]*>/ /g' | wc -w
curl -s https://yoursite.com | grep -o "<h1[^>]*>[^<]*"
curl -s https://yoursite.com/robots.txt
curl -s https://yoursite.com | grep -c "what you actually sell"
Four commands. If all four look wrong, that is not an SEO problem to schedule for next quarter. It is the site not being readable, and it is usually a much smaller fix than a redesign.
Run it on your own site before you run it on a client's. I have been surprised more than once.