Your page looks perfect in Chrome. Headline, pricing, FAQ, a nice comparison table. But an AI crawler doesn't open Chrome. It sends one HTTP request, gets back whatever your server returns, and moves on.
If your important text only appears after JavaScript runs, there's a real chance that crawler saw an empty shell. This post shows how to check that in about ten minutes with curl and a couple of one-liners.
There are two versions of every page:
Google documents a separate rendering stage for JavaScript (JavaScript SEO basics). Other clients don't have to copy Google's capabilities. In December 2024, Vercel and MERJ published an analysis of AI crawler traffic on Vercel's network (The rise of the AI crawler). They found that OpenAI's crawlers, ClaudeBot, PerplexityBot and several others fetched JavaScript files but didn't execute them, while Gemini (via Google's infrastructure) and Applebot did render pages. Crawler behavior can change over time, so treat this as a reason to test, not as a permanent rule.
The safe assumption: if a fact isn't in the initial HTML, don't count on an AI crawler seeing it.
Don't start with a full crawl. Choose:
For each one, write down 3-5 facts that must be visible: the H1, the main description, a price or condition, a key claim, and the important internal links.
URL="https://example.com/pricing"
curl -sL "$URL" -o raw.html
wc -c raw.html
Ignore the file size. A 300 KB response can be all navigation and inline JSON; a 20 KB response can contain the full answer. What matters is meaning.
To see the response a specific crawler would get, you can also send its user agent string (useful when a CDN or WAF treats bots differently):
curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
-o raw_gptbot.html -w "%{http_code}\n" "$URL"
If this returns 403, 429 or a challenge page while a normal request returns 200, you have a different problem: your firewall, not your rendering. Fix that first.
for phrase in "Pricing" "per month" "Free trial" "How it works"; do
if grep -qi "$phrase" raw.html; then
echo "FOUND: $phrase"
else
echo "MISSING: $phrase"
fi
done
A match isn't the whole story. Text sitting inside a JSON blob in a <script> tag (for example __NEXT_DATA__ or a hydration payload) is technically "in the HTML," but a normal content extractor may not treat it as the main content. (The Vercel study notes that models can sometimes interpret JSON in the initial response, but you shouldn't rely on it.) Check where the match lives:
grep -o -i ".\{80\}per month.\{80\}" raw.html | head
If every hit is inside a script, mark it as "present in data, not in markup."
A quick way to see the visible text without a browser:
python3 - <<'PY'
import re, html
t = open("raw.html", encoding="utf-8", errors="ignore").read()
t = re.sub(r"(?is)<(script|style|noscript|svg)[^>]*>.*?</\1>", " ", t)
t = re.sub(r"(?s)<[^>]+>", " ", t)
t = re.sub(r"\s+", " ", html.unescape(t)).strip()
print(len(t.split()), "words")
print(t[:1500])
PY
Read the first 1,500 characters. Does it say what the page is about and what you offer? Or is it "... Menu Sign in Cookie settings"?
Open the same URL in your browser, open DevTools and compare the rendered DOM with raw.html. Note which important elements only exist after load: tabs, accordions filled by API calls, pricing tables from a widget, reviews from a third-party script.
Also test direct URL access. Single-page apps sometimes work when you click from the homepage but return a generic shell (or a 404) when the route is requested directly. Crawlers always request directly.
If the useful text is missing from the initial HTML, the usual options are:
Don't build a separate "bot version" with different claims. Visitors and crawlers should get the same meaning; serving different content to bots is the kind of thing that backfires.
When the fix ships, re-run the same curl and the same checks. Keep both files:
curl -sL "$URL" -o "after_$(date +%F).html"
A fix is done when the main text is present in a direct request, the page still works for humans, and the required links are there. That removes a technical barrier. It doesn't make the page relevant to every question or guarantee a citation, so track discovery and citations separately.
If you want this check done automatically, AURA GEO audits the HTML your server actually returns, flags pages where little readable text comes through, and turns that into tasks you can hand to your own agent over MCP. There's a longer explainer here: how to tell whether a crawler can see the text on a JavaScript website.