{"slug": "curl-your-own-homepage-that-is-all-chatgpt-sees", "title": "curl your own homepage. That is all ChatGPT sees.", "summary": "A developer warns that many AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, do not execute JavaScript, making client-rendered websites invisible to them. The post advises using curl to check if a site's content is present in raw HTML and recommends server-side rendering or prerendering to ensure visibility to AI assistants.", "body_md": "Run this against your site right now:\n\n```\ncurl -s https://yoursite.com | grep -o \"<h1[^>]*>.*</h1>\"\n```\n\nIf nothing comes back, or you get an empty `<div id=\"root\">`\n\n, then large parts of the internet cannot read your site. Not \"reads it poorly\". Cannot read it.\n\nI do this on every site we take over, and the result surprises people often enough that it is worth writing down.\n\n`curl`\n\ndoes exactly one thing: it fetches HTML and stops. It does not run JavaScript. It does not wait for hydration. It does not call your API.\n\nThat is also what a large number of crawlers do.\n\nGooglebot is the exception people think of, and it is genuinely good: it fetches, queues the page, and renders it with a headless browser later. Client rendered content usually gets indexed eventually.\n\nThe AI crawlers are a different story. As of now, the major ones (GPTBot, ClaudeBot, PerplexityBot, and friends) largely do not execute JavaScript. They fetch the HTML, take what is in it, and move on. Whatever your framework paints after the bundle loads is invisible to them.\n\nSo `curl`\n\nis a decent proxy for the floor: if your content is not in that response, assume a meaningful slice of automated readers never see it.\n\nFor years the bet was reasonable. Google renders JS, Google is search, so client rendering was survivable.\n\nThen a chunk of discovery moved to assistants. People ask ChatGPT for a recommendation instead of scrolling ten blue links. If the model cannot read your page, you are not in the answer, and there is no page two to be on.\n\nFor a marketing site this is the whole ballgame. For a small business it is worse, because the queries that matter (\"web designers in X\", \"who does Y near me\") are precisely the ones people now ask an assistant.\n\n**1. Raw HTML, by word count.**\n\n```\ncurl -s https://yoursite.com | wc -c        # total bytes\ncurl -s https://yoursite.com | \\\n  sed 's/<script[^>]*>.*<\\/script>//g' | \\\n  sed 's/<[^>]*>/ /g' | wc -w               # actual visible words\n```\n\nA marketing homepage with 40 words of real text in the raw HTML is a red flag. A React SPA often returns fewer than 10.\n\n**2. Compare raw against rendered.**\n\n```\n// in DevTools console, on your live site\ndocument.body.innerText.split(/\\s+/).length\n```\n\nCompare that number to the `wc -w`\n\nabove. A large gap is the amount of your site that only exists after JavaScript runs.\n\n**3. Ask with the right user agent.**\n\n```\ncurl -s -A \"GPTBot\" https://yoursite.com | grep -c \"your-key-phrase\"\n```\n\nAlso worth confirming you are not blocking these crawlers by accident. Plenty of `robots.txt`\n\nfiles quietly disallow them because someone pasted a blocklist from a blog post in 2023:\n\n```\nUser-agent: GPTBot\nDisallow: /\n```\n\nThat is a choice you can make deliberately. Making it by accident is expensive.\n\nThe fix is not \"switch frameworks\". It is \"make sure the words are in the first response\".\n\n**Next.js App Router.** Server Components are the default, so you are mostly fine until someone adds `\"use client\"`\n\nat the top of a page to use one hook. Push `\"use client\"`\n\ndown to the leaf that needs it instead of the page.\n\n```\n// page.tsx stays a server component\nexport default async function Page() {\n  const data = await getData();\n  return (\n    <>\n      <h1>{data.title}</h1>\n      <p>{data.summary}</p>\n      <InteractiveWidget />   {/* only this is \"use client\" */}\n    </>\n  );\n}\n```\n\nThe heading and copy are now in the HTML. The widget hydrates after. Both readers are served.\n\n**Vite, CRA, plain SPA.** You need prerendering. `vite-plugin-ssr`\n\n/ Vike, or a prerender step for known routes, or move the marketing pages to static HTML and keep the SPA for the app itself. The split is usually the honest answer: your dashboard does not need to be crawlable, your pricing page does.\n\n**Astro / Eleventy / Hugo.** Already static. Check anyway, because a client only island can still swallow your main content if someone got enthusiastic.\n\n**Anything behind a cookie banner that blanks the page.** Some consent implementations render an interstitial and nothing else until a click. Crawlers do not click.\n\nGetting the HTML right is necessary but it is not sufficient. Assistants quote text. Give them text worth quoting.\n\n`<h1>`\n\nthat states what you do, in the words a customer would use, not a slogan.We rebuilt a manufacturer's site last year where the entire product range lived inside a JavaScript carousel. The raw HTML contained the company name and a cookie notice. Moving the product copy into server rendered markup was most of the work, and it was not a redesign, it was a text problem wearing a framework costume.\n\n```\n# 1. Is there content at all?\ncurl -s https://yoursite.com | sed 's/<[^>]*>/ /g' | wc -w\n\n# 2. Is the h1 there?\ncurl -s https://yoursite.com | grep -o \"<h1[^>]*>[^<]*\"\n\n# 3. Are you blocking the crawlers?\ncurl -s https://yoursite.com/robots.txt\n\n# 4. Does a key phrase survive?\ncurl -s https://yoursite.com | grep -c \"what you actually sell\"\n```\n\nFour commands. If all four look wrong, that is not an SEO problem to schedule for next quarter. It is the site not being readable, and it is usually a much smaller fix than a redesign.\n\nRun it on your own site before you run it on a client's. I have been surprised more than once.", "url": "https://wpnews.pro/news/curl-your-own-homepage-that-is-all-chatgpt-sees", "canonical_source": "https://dev.to/techchoom/curl-your-own-homepage-that-is-all-chatgpt-sees-3kpm", "published_at": "2026-08-29 15:36:27+00:00", "updated_at": "2026-08-29 15:49:05.156052+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "ai-infrastructure"], "entities": ["GPTBot", "ClaudeBot", "PerplexityBot", "Googlebot", "Next.js", "Vite", "Astro", "Eleventy"], "alternates": {"html": "https://wpnews.pro/news/curl-your-own-homepage-that-is-all-chatgpt-sees", "markdown": "https://wpnews.pro/news/curl-your-own-homepage-that-is-all-chatgpt-sees.md", "text": "https://wpnews.pro/news/curl-your-own-homepage-that-is-all-chatgpt-sees.txt", "jsonld": "https://wpnews.pro/news/curl-your-own-homepage-that-is-all-chatgpt-sees.jsonld"}}