{"slug": "how-to-check-whether-ai-search-can-read-your-website", "title": "How to check whether AI search can read your website", "summary": "The maintainer of GEO Doctor released an MIT-licensed checker that diagnoses why AI search crawlers cannot read a website, testing four conditions: robots.txt rules blocking search crawlers such as OAI-SearchBot and PerplexityBot, noindex tags, client-rendered text invisible without JavaScript, and firewall or CDN blocks. The tool runs from a Node.js 18+ command line or inside AI assistants like Claude Code and Codex, keeps data in local plain files with no server, and can generate a fact page, FAQ, llms.txt and structured data hosted on GitHub Pages.", "body_md": "Check four things: whether robots.txt blocks AI search crawlers, whether the page carries noindex, whether the main text is visible without running JavaScript, and whether crawlers are blocked by a firewall or CDN. GEO Doctor (geo-doctor) has a free, MIT-licensed checker for these.\n\n## \n  \n  \n  Why AI search may not see your site\n\nAI assistants that search the web fetch pages first and answer from what they fetched. If they cannot fetch your page, or get an empty shell, they will not cite you. Four usual causes:\n\n1. \n**robots.txt blocks AI search crawlers.** There are two kinds of AI crawler. Search crawlers such as OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot fetch pages to answer questions; blocking them hides you from AI search. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect training data; blocking them usually does not affect AI search. Many templates and security plugins block both kinds at once, so look at them separately.\n2. \n**noindex.** A noindex meta tag or header makes search engines and some AI search skip the page.\n3. \n**Client-rendered text.** If the HTML is nearly empty until JavaScript runs, fetchers that do not run scripts see a blank page.\n4. \n**Firewall or CDN rules** that treat crawlers as attacks.\n\n## \n  \n  \n  Checking with GEO Doctor\n\nGEO Doctor checks robots.txt, AI crawlers, noindex and whether the main text is visible, and tells you which item fails. It checks technical readability only, not whether the content is good. It runs from a Node.js 18+ command line or inside AI assistants such as Claude Code, Codex and Kimi Code. Your data stays in plain files in your own project; GEO Doctor has no server.\n\n## \n  \n  \n  Fixing what you find\n\n- robots.txt: allow the search crawlers if you want AI search to read you. Whether to allow training crawlers is a separate choice.\n- noindex: check whether a staging setting leaked into production.\n- Invisible text: serve key content in the initial HTML, or publish static versions of important pages.\n- If you want a page written for AI, GEO Doctor can turn your fact card into a fact site (fact page, FAQ, llms.txt, structured data) hosted free on GitHub Pages.\n\nBeing readable is only step one: AI also needs clear, consistent content about you to cite.\n\nSource: [https://github.com/Fangx-AI/geo-doctor](https://github.com/Fangx-AI/geo-doctor)\n\nDisclosure: I maintain GEO Doctor, and this article was drafted with AI assistance (Claude) and reviewed by me.", "url": "https://wpnews.pro/news/how-to-check-whether-ai-search-can-read-your-website", "canonical_source": "https://dev.to/fangxai/how-to-check-whether-ai-search-can-read-your-website-566", "published_at": "2026-10-20 05:13:12+00:00", "updated_at": "2026-10-20 05:17:12.293817+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "structured-data", "developer-tools"], "entities": ["GEO Doctor", "OAI-SearchBot", "PerplexityBot", "GPTBot", "ClaudeBot", "Google-Extended", "CCBot", "GitHub Pages"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-check-whether-ai-search-can-read-your-website", "markdown": "https://wpnews.pro/news/how-to-check-whether-ai-search-can-read-your-website.md", "text": "https://wpnews.pro/news/how-to-check-whether-ai-search-can-read-your-website.txt", "jsonld": "https://wpnews.pro/news/how-to-check-whether-ai-search-can-read-your-website.jsonld"}}