{"slug": "can-ai-crawlers-see-your-javascript-a-10-minute-curl-test-for-the-html-bots-get", "title": "Can AI Crawlers See Your JavaScript? A 10-Minute curl Test for the HTML Bots Actually Get", "summary": "A developer published a 10-minute curl-based test for checking whether AI crawlers can see page content that only appears after JavaScript runs. The method fetches a URL with curl, greps for key facts like headlines and prices, and strips scripts to reveal the visible text, based on a December 2024 Vercel and MERJ analysis finding that OpenAI's crawlers, ClaudeBot and PerplexityBot fetched JavaScript files without executing them, while Gemini and Applebot did render pages.", "body_md": "Your page looks perfect in Chrome. Headline, pricing, FAQ, a nice comparison table. But an AI crawler doesn't open Chrome. It sends one HTTP request, gets back whatever your server returns, and moves on.\n\nIf your important text only appears after JavaScript runs, there's a real chance that crawler saw an empty shell. This post shows how to check that in about ten minutes with `curl` and a couple of one-liners.\n\nThere are two versions of every page:\n\nGoogle documents a separate rendering stage for JavaScript ([JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)). Other clients don't have to copy Google's capabilities. In December 2024, Vercel and MERJ published an analysis of AI crawler traffic on Vercel's network ([The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler)). They found that OpenAI's crawlers, ClaudeBot, PerplexityBot and several others fetched JavaScript files but didn't execute them, while Gemini (via Google's infrastructure) and Applebot did render pages. Crawler behavior can change over time, so treat this as a reason to test, not as a permanent rule.\n\nThe safe assumption: **if a fact isn't in the initial HTML, don't count on an AI crawler seeing it.**\n\nDon't start with a full crawl. Choose:\n\nFor each one, write down 3-5 facts that must be visible: the H1, the main description, a price or condition, a key claim, and the important internal links.\n\n```\nURL=\"https://example.com/pricing\"\n\ncurl -sL \"$URL\" -o raw.html\nwc -c raw.html\n```\n\nIgnore the file size. A 300 KB response can be all navigation and inline JSON; a 20 KB response can contain the full answer. What matters is meaning.\n\nTo see the response a specific crawler would get, you can also send its user agent string (useful when a CDN or WAF treats bots differently):\n\n```\ncurl -sL -A \"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)\" \\\n  -o raw_gptbot.html -w \"%{http_code}\\n\" \"$URL\"\n```\n\nIf this returns `403`, `429` or a challenge page while a normal request returns `200`, you have a different problem: your firewall, not your rendering. Fix that first.\n\n```\nfor phrase in \"Pricing\" \"per month\" \"Free trial\" \"How it works\"; do\n  if grep -qi \"$phrase\" raw.html; then\n    echo \"FOUND:   $phrase\"\n  else\n    echo \"MISSING: $phrase\"\n  fi\ndone\n```\n\nA match isn't the whole story. Text sitting inside a JSON blob in a `<script>` tag (for example `__NEXT_DATA__` or a hydration payload) is technically \"in the HTML,\" but a normal content extractor may not treat it as the main content. (The Vercel study notes that models can sometimes interpret JSON in the initial response, but you shouldn't rely on it.) Check where the match lives:\n\n```\ngrep -o -i \".\\{80\\}per month.\\{80\\}\" raw.html | head\n```\n\nIf every hit is inside a script, mark it as \"present in data, not in markup.\"\n\nA quick way to see the visible text without a browser:\n\n``` python\npython3 - <<'PY'\nimport re, html\nt = open(\"raw.html\", encoding=\"utf-8\", errors=\"ignore\").read()\nt = re.sub(r\"(?is)<(script|style|noscript|svg)[^>]*>.*?</\\1>\", \" \", t)\nt = re.sub(r\"(?s)<[^>]+>\", \" \", t)\nt = re.sub(r\"\\s+\", \" \", html.unescape(t)).strip()\nprint(len(t.split()), \"words\")\nprint(t[:1500])\nPY\n```\n\nRead the first 1,500 characters. Does it say what the page is about and what you offer? Or is it \"Loading... Menu Sign in Cookie settings\"?\n\nOpen the same URL in your browser, open DevTools and compare the rendered DOM with `raw.html`. Note which important elements only exist after load: tabs, accordions filled by API calls, pricing tables from a widget, reviews from a third-party script.\n\nAlso test **direct URL access**. Single-page apps sometimes work when you click from the homepage but return a generic shell (or a 404) when the route is requested directly. Crawlers always request directly.\n\nIf the useful text is missing from the initial HTML, the usual options are:\n\nDon't build a separate \"bot version\" with different claims. Visitors and crawlers should get the same meaning; serving different content to bots is the kind of thing that backfires.\n\nWhen the fix ships, re-run the same `curl` and the same checks. Keep both files:\n\n```\ncurl -sL \"$URL\" -o \"after_$(date +%F).html\"\n```\n\nA fix is done when the main text is present in a direct request, the page still works for humans, and the required links are there. That removes a technical barrier. It doesn't make the page relevant to every question or guarantee a citation, so track discovery and citations separately.\n\nIf you want this check done automatically, [AURA GEO](https://aurageo.io/en/) audits the HTML your server actually returns, flags pages where little readable text comes through, and turns that into tasks you can hand to your own agent over MCP. There's a longer explainer here: [how to tell whether a crawler can see the text on a JavaScript website](https://aurageo.io/en/learn/html-javascript-i-chtenie-sajta-ii/).", "url": "https://wpnews.pro/news/can-ai-crawlers-see-your-javascript-a-10-minute-curl-test-for-the-html-bots-get", "canonical_source": "https://dev.to/aurageo/can-ai-crawlers-see-your-javascript-a-10-minute-curl-test-for-the-html-bots-actually-get-2j35", "published_at": "2026-10-22 16:09:01+00:00", "updated_at": "2026-10-22 16:13:31.197915+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "developer-tools"], "entities": ["Vercel", "MERJ", "OpenAI", "GPTBot", "ClaudeBot", "PerplexityBot", "Google", "Applebot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/can-ai-crawlers-see-your-javascript-a-10-minute-curl-test-for-the-html-bots-get", "markdown": "https://wpnews.pro/news/can-ai-crawlers-see-your-javascript-a-10-minute-curl-test-for-the-html-bots-get.md", "text": "https://wpnews.pro/news/can-ai-crawlers-see-your-javascript-a-10-minute-curl-test-for-the-html-bots-get.txt", "jsonld": "https://wpnews.pro/news/can-ai-crawlers-see-your-javascript-a-10-minute-curl-test-for-the-html-bots-get.jsonld"}}