Can AI Crawlers See Your JavaScript? A 10-Minute curl Test for the HTML Bots Actually Get

A developer published a 10-minute curl-based test for checking whether AI crawlers can see page content that only appears after JavaScript runs. The method fetches a URL with curl, greps for key facts like headlines and prices, and strips scripts to reveal the visible text, based on a December 2024 Vercel and MERJ analysis finding that OpenAI's crawlers, ClaudeBot and PerplexityBot fetched JavaScript files without executing them, while Gemini and Applebot did render pages.

Your page looks perfect in Chrome. Headline, pricing, FAQ, a nice comparison table. But an AI crawler doesn't open Chrome. It sends one HTTP request, gets back whatever your server returns, and moves on. If your important text only appears after JavaScript runs, there's a real chance that crawler saw an empty shell. This post shows how to check that in about ten minutes with curl and a couple of one-liners. There are two versions of every page: Google documents a separate rendering stage for JavaScript JavaScript SEO basics https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics . Other clients don't have to copy Google's capabilities. In December 2024, Vercel and MERJ published an analysis of AI crawler traffic on Vercel's network The rise of the AI crawler https://vercel.com/blog/the-rise-of-the-ai-crawler . They found that OpenAI's crawlers, ClaudeBot, PerplexityBot and several others fetched JavaScript files but didn't execute them, while Gemini via Google's infrastructure and Applebot did render pages. Crawler behavior can change over time, so treat this as a reason to test, not as a permanent rule. The safe assumption: if a fact isn't in the initial HTML, don't count on an AI crawler seeing it. Don't start with a full crawl. Choose: For each one, write down 3-5 facts that must be visible: the H1, the main description, a price or condition, a key claim, and the important internal links. URL="https://example.com/pricing" curl -sL "$URL" -o raw.html wc -c raw.html Ignore the file size. A 300 KB response can be all navigation and inline JSON; a 20 KB response can contain the full answer. What matters is meaning. To see the response a specific crawler would get, you can also send its user agent string useful when a CDN or WAF treats bots differently : curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot " \ -o raw gptbot.html -w "%{http code}\n" "$URL" If this returns 403 , 429 or a challenge page while a normal request returns 200 , you have a different problem: your firewall, not your rendering. Fix that first. for phrase in "Pricing" "per month" "Free trial" "How it works"; do if grep -qi "$phrase" raw.html; then echo "FOUND: $phrase" else echo "MISSING: $phrase" fi done A match isn't the whole story. Text sitting inside a JSON blob in a <script tag for example NEXT DATA or a hydration payload is technically "in the HTML," but a normal content extractor may not treat it as the main content. The Vercel study notes that models can sometimes interpret JSON in the initial response, but you shouldn't rely on it. Check where the match lives: grep -o -i ".\{80\}per month.\{80\}" raw.html | head If every hit is inside a script, mark it as "present in data, not in markup." A quick way to see the visible text without a browser: python python3 - <<'PY' import re, html t = open "raw.html", encoding="utf-8", errors="ignore" .read t = re.sub r" ?is < script|style|noscript|svg ^ . ?</\1 ", " ", t t = re.sub r" ?s < ^ + ", " ", t t = re.sub r"\s+", " ", html.unescape t .strip print len t.split , "words" print t :1500 PY Read the first 1,500 characters. Does it say what the page is about and what you offer? Or is it "Loading... Menu Sign in Cookie settings"? Open the same URL in your browser, open DevTools and compare the rendered DOM with raw.html . Note which important elements only exist after load: tabs, accordions filled by API calls, pricing tables from a widget, reviews from a third-party script. Also test direct URL access . Single-page apps sometimes work when you click from the homepage but return a generic shell or a 404 when the route is requested directly. Crawlers always request directly. If the useful text is missing from the initial HTML, the usual options are: Don't build a separate "bot version" with different claims. Visitors and crawlers should get the same meaning; serving different content to bots is the kind of thing that backfires. When the fix ships, re-run the same curl and the same checks. Keep both files: curl -sL "$URL" -o "after $ date +%F .html" A fix is done when the main text is present in a direct request, the page still works for humans, and the required links are there. That removes a technical barrier. It doesn't make the page relevant to every question or guarantee a citation, so track discovery and citations separately. If you want this check done automatically, AURA GEO https://aurageo.io/en/ audits the HTML your server actually returns, flags pages where little readable text comes through, and turns that into tasks you can hand to your own agent over MCP. There's a longer explainer here: how to tell whether a crawler can see the text on a JavaScript website https://aurageo.io/en/learn/html-javascript-i-chtenie-sajta-ii/ .