{"slug": "the-ai-crawler-checklist-why-the-model-never-even-read-your-page", "title": "The AI Crawler Checklist: Why the Model Never Even Read Your Page", "summary": "An engineer's checklist reveals why AI assistants often fail to cite web pages: crawlers like GPTBot, ClaudeBot, and PerplexityBot may be blocked by robots.txt or CDN settings, or pages may rely on client-side rendering that crawlers can't execute. The post advises testing raw HTML, allowing AI bots, using server-side rendering, and adding llms.txt to improve visibility.", "body_md": "You can write the perfect page, answer-first, honest, quotable, and still get zero AI citations. Not because the content lost. Because the crawler never got in.\n\nAI assistants don't read the web live for every answer. They rely on crawlers, GPTBot, ClaudeBot, PerplexityBot, Google's crawlers, that fetch and store your pages so a model can retrieve them later. If those bots can't reach or render your page, you're invisible before the content ever gets judged. This is the plumbing nobody checks.\n\nHere's the full checklist. Run it on your own site today.\n\nThe most common own-goal. Open your `robots.txt`\n\n(it's at `yoursite.com/robots.txt`\n\n) and look for lines like this:\n\n```\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n```\n\nA lot of sites added these in 2023 when blocking AI training was the trend. The problem: the same bots that \"train\" also fetch pages to answer live questions. Block them and you opt out of being cited at all. Decide on purpose. If you want AI visibility, let them in:\n\n```\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n```\n\nCheck your CDN too. Cloudflare and others ship a one-click \"block AI bots\" toggle that overrides your `robots.txt`\n\n. I've seen founders swear their site was open while Cloudflare quietly returned a 403 to every AI crawler. Test the real response, not the setting.\n\nThis one kills more vibe-coded sites than anything else. Many crawlers fetch your raw HTML and don't run JavaScript, or run a limited version of it. If your page ships an empty `<div id=\"root\">`\n\nand paints all the real text with client-side React, the crawler sees a blank page.\n\nTest it in ten seconds. Open your page, view source (Ctrl+U), and search for a full sentence from your main content. If it's in the raw HTML, you're fine. If the source is just script tags and an empty shell, the crawler probably sees nothing.\n\nThe fix is server-side rendering or static generation, so the words are in the HTML on first load. If you're on Next.js, Nuxt, Astro or plain HTML, you're likely fine. If you're on a pure client-side single-page app, this is your biggest leak.\n\nCrawlers and models favor pages that answer fast. If your real content sits below a hero section, a cookie banner, three testimonials and a newsletter box, the model has to dig for the passage worth quoting, and often it just moves to a cleaner source.\n\nPut a direct, quotable answer near the top. One clear paragraph that responds to the question the page targets, before any preamble. Write it so it can be lifted whole into an AI answer with nothing else attached.\n\nModels parse structure to understand and extract. Give them clear signals:\n\n↳ One `<h1>`\n\nthat states the topic, then `<h2>`\n\ns that each match a real follow-up question.\n\n↳ Real lists and tables in HTML, not screenshots of lists. A model can quote a `<table>`\n\n. It can't read a PNG.\n\n↳ Short paragraphs. Dense walls of text are harder to extract a clean passage from.\n\n↳ A visible last-updated date. Freshness is a trust signal, and models lean toward current pages.\n\n`llms.txt`\n\nis an emerging standard: a plain-text file at your root (`yoursite.com/llms.txt`\n\n) that hands AI systems a clean map of your most important pages, in Markdown, without the nav, ads and clutter of your full site.\n\nIt's not yet honored by every model, and it won't rescue a page the crawler is otherwise blocked from. Treat it as cheap insurance, not a magic switch. A minimal one looks like this:\n\n```\n# Your Brand\n\n> One sentence on what you do.\n\n## Core pages\n- [What is [category]](https://yoursite.com/what-is): the definitive explainer\n- [Your product vs alternatives](https://yoursite.com/compare): honest comparison\n- [Pricing](https://yoursite.com/pricing): plans and what each includes\n```\n\nTen minutes to write, and it makes your best pages easy for a model to find and quote. There's little downside.\n\nA page nobody has crawled can't be cited. Check that your key pages are submitted in a sitemap, aren't marked `noindex`\n\nby accident, and return a clean 200, not a redirect chain or a soft 404. One stray `noindex`\n\nmeta tag left over from a staging build can hide a page for months.\n\nSteps 1 and 2, the blocked-crawler and the JavaScript-shell problems, are the two that silently sink the most sites, and they're annoying to test properly across every bot.\n\nSo we built a free tool that does it for you. Drop your URL into the [AI crawler check at fulcru.app/tools/ai-crawler-check](https://fulcru.app/tools/ai-crawler-check) and it fetches your page the way GPTBot, ClaudeBot and PerplexityBot do, then tells you what each one actually sees: whether you're blocked, whether your content renders without JavaScript, and whether the answer is reachable. No card, no signup. It's the fastest way to catch the leak that's making all your good content invisible.\n\nContent strategy gets all the attention, but it sits on top of the plumbing. Fix the plumbing first:\n\nLet the AI crawlers in.\n\nMake sure your words are in the raw HTML.\n\nAnswer in the first 100 words.\n\nGive the model clean structure to extract.\n\nAdd an llms.txt as cheap insurance.\n\nConfirm your pages are indexed.\n\nDo this and every page you write afterward has a chance to be read. Skip it and the best AEO content in your category never gets seen by the one reader you're writing for.\n\nRun the crawler check today. It takes 30 seconds and it might explain a silence you've been blaming on your content.\n\n*Want to see which questions ChatGPT, Gemini and Perplexity already answer with your competitor's name? Run the free Fulcru visibility report at fulcru.app.*", "url": "https://wpnews.pro/news/the-ai-crawler-checklist-why-the-model-never-even-read-your-page", "canonical_source": "https://dev.to/stefan_vasile_1ad9799e929/the-ai-crawler-checklist-why-the-model-never-even-read-your-page-5en5", "published_at": "2026-08-15 16:59:52+00:00", "updated_at": "2026-08-15 17:11:57.177406+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "developer-tools", "large-language-models"], "entities": ["GPTBot", "ClaudeBot", "PerplexityBot", "Cloudflare", "Next.js", "Nuxt", "Astro"], "alternates": {"html": "https://wpnews.pro/news/the-ai-crawler-checklist-why-the-model-never-even-read-your-page", "markdown": "https://wpnews.pro/news/the-ai-crawler-checklist-why-the-model-never-even-read-your-page.md", "text": "https://wpnews.pro/news/the-ai-crawler-checklist-why-the-model-never-even-read-your-page.txt", "jsonld": "https://wpnews.pro/news/the-ai-crawler-checklist-why-the-model-never-even-read-your-page.jsonld"}}