{"slug": "i-made-a-claude-code-skill-to-check-if-ai-crawlers-can-read-your-site", "title": "I made a Claude Code skill to check if AI crawlers can read your site", "summary": "A developer released a Claude Code skill that diagnoses whether websites are visible to AI crawlers like ClaudeBot, GPTBot, and PerplexityBot, which do not run JavaScript. The tool fetches pages as raw HTML, reports verdicts of visible, partial, or invisible, and provides framework-specific fixes. It aims to help sites that rank on Google but remain invisible in AI search results due to client-side rendering.", "body_md": "**A Claude Code skill that tells you whether your website is actually visible to AI search — and exactly how to fix it if it isn't.**\n\nChatGPT Search, Claude, and Perplexity are sending people to websites now. But the crawlers that feed them — **ClaudeBot, GPTBot, PerplexityBot, Bytespider, Meta-ExternalAgent** — *do not run JavaScript*. They read only the raw HTML your server returns.\n\nGoogle's renderer has handled JavaScript for years, so a client-rendered site (React, Vite, Vue, Svelte, any SPA) can rank perfectly fine on Google and still be **completely invisible in AI answers**.\n\nIf you've ever thought *\"we get Google traffic but nothing from AI\"* — or you shipped a slick vibe-coded SPA and you're nowhere in ChatGPT — this is almost always why.\n\nThis skill fetches your page(s) the same way a non-JS crawler does — raw HTML, no browser — and reports, per page, whether your real content and key signals (`title`\n\n, meta description, canonical, Open Graph, JSON-LD, crawlable links) are actually present.\n\nThen it prescribes a **concrete, framework-specific fix**. Not \"consider SSR\" — the exact tool, config change, and one-line way to verify it worked.\n\nEvery page gets one of three verdicts:\n\n| Verdict | What it means |\n|---|---|\n✅ VISIBLE |\nReal content is in the raw HTML. The crawler can read you. |\nPARTIAL |\nOnly a fraction is in raw HTML; the main body is JS-rendered. |\n❌ INVISIBLE |\nThe crawler gets an empty shell (`<div id=\"root\"></div>` ). Nothing useful is there. |\n\nIt **diagnoses and prescribes**. It does not write marketing copy, do keyword research, or chase backlinks.\n\nThis is a [Claude Code Agent Skill](https://docs.claude.com/en/docs/claude-code/overview). Drop the folder into your skills directory and Claude Code picks it up automatically.\n\n**Personal (available in every project):**\n\n```\ngit clone https://github.com/<merqrylabs>/ai-crawler-visibility.git \\\n  ~/.claude/skills/ai-crawler-visibility\n```\n\n**Project-scoped (just this repo):**\n\n```\ngit clone https://github.com/<merqrylabs>/ai-crawler-visibility.git \\\n  ./.claude/skills/ai-crawler-visibility\n```\n\nThat's it. No npm install, no API keys, no third-party packages — the analyzer runs on plain Python 3.\n\nCheck the official\n\n[Agent Skills docs]for the canonical skills path if your setup differs.\n\nJust talk to Claude Code in natural language. The skill triggers on its own when you ask the kind of question it's built for:\n\n*\"Why isn't my site showing up in ChatGPT?\"**\"Is my new site visible to AI search engines?\"**\"I added JSON-LD but rich results still aren't showing.\"*\n\nYou can also run the bundled analyzer directly:\n\n```\n# Single page\npython3 scripts/check_visibility.py https://thesite.com\n\n# Several specific pages\npython3 scripts/check_visibility.py https://thesite.com https://thesite.com/pricing\n\n# Sample from a sitemap (checks up to --limit pages)\npython3 scripts/check_visibility.py --sitemap https://thesite.com/sitemap.xml --limit 10\n```\n\nAdd `--json`\n\nto process results programmatically.\n\n⚠️ Always check yourproductionURL, not`localhost`\n\n— local dev servers don't reflect how the deployed site is served to crawlers.\n\n```\n# AI Crawler Visibility Report\n\n## Verdict\nhttps://thesite.com         — INVISIBLE\nhttps://thesite.com/pricing — INVISIBLE\n\n## What the crawler actually sees\n/ : 0 visible words. Missing from raw HTML: meta description,\n    canonical, Open Graph, JSON-LD, crawlable links.\n\n## Fixes (highest impact first)\n[Concrete, framework-specific steps — each with how to verify]\n\n## How to confirm the fix\nView Page Source (not Inspect) and confirm your headline text\nand JSON-LD appear in the raw HTML.\n```\n\nThis skill tells you *what's broken* and *what the fix is*. If you'd rather not hand-implement prerendering and static JSON-LD injection yourself, the companion **AI Crawler Fixer** pack does it for you:\n\n- Injects static SEO + JSON-LD straight into\n`index.html`\n\n- Sets up post-build prerendering so real content lands in the raw HTML\n- Works through approval gates so nothing touches your codebase without your sign-off\n- Re-runs the diagnosis after to prove the fix landed\n\n👉 **[Get the AI Crawler Fixer pack →] [https://novae8.gumroad.com/l/AICrawlerFixer](https://novae8.gumroad.com/l/AICrawlerFixer)\n\n*Diagnosis is free and always will be. The Fixer is for when you want the problem gone today.*\n\n**The User-Agent isn't what makes a page invisible — not executing JavaScript is.**(The script still sends a real bot UA as a second request to detect server-side prerendering.)** JSON-LD injected by JavaScript is invisible**to non-rendering crawlers. It has to be in the raw HTML.** Hydration is fine.**It attaches handlers to already-rendered HTML, so the content's already there. Pure client-side rendering (CSR) is the problem.**\"View Page Source\"** shows raw HTML (what the crawler sees).**\"Inspect\"** shows the rendered DOM (what it does*not*see). Always verify with View Page Source.\n\nMIT — use it, fork it, ship it.\n\nBuilt by [Novae Systems](https://getnovaesystems.com) — websites, AI automation, and SEO for local businesses.\n\n*Novae builds. Systems flow.*", "url": "https://wpnews.pro/news/i-made-a-claude-code-skill-to-check-if-ai-crawlers-can-read-your-site", "canonical_source": "https://github.com/MerqryLabs/ai-crawler-visibility", "published_at": "2026-06-26 06:42:49+00:00", "updated_at": "2026-06-26 07:04:38.951584+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-products"], "entities": ["Claude Code", "ClaudeBot", "GPTBot", "PerplexityBot", "Bytespider", "Meta-ExternalAgent", "Google", "ChatGPT"], "alternates": {"html": "https://wpnews.pro/news/i-made-a-claude-code-skill-to-check-if-ai-crawlers-can-read-your-site", "markdown": "https://wpnews.pro/news/i-made-a-claude-code-skill-to-check-if-ai-crawlers-can-read-your-site.md", "text": "https://wpnews.pro/news/i-made-a-claude-code-skill-to-check-if-ai-crawlers-can-read-your-site.txt", "jsonld": "https://wpnews.pro/news/i-made-a-claude-code-skill-to-check-if-ai-crawlers-can-read-your-site.jsonld"}}