The AI Crawler Checklist: Why the Model Never Even Read Your Page

An engineer's checklist reveals why AI assistants often fail to cite web pages: crawlers like GPTBot, ClaudeBot, and PerplexityBot may be blocked by robots.txt or CDN settings, or pages may rely on client-side rendering that crawlers can't execute. The post advises testing raw HTML, allowing AI bots, using server-side rendering, and adding llms.txt to improve visibility.

You can write the perfect page, answer-first, honest, quotable, and still get zero AI citations. Not because the content lost. Because the crawler never got in. AI assistants don't read the web live for every answer. They rely on crawlers, GPTBot, ClaudeBot, PerplexityBot, Google's crawlers, that fetch and store your pages so a model can retrieve them later. If those bots can't reach or render your page, you're invisible before the content ever gets judged. This is the plumbing nobody checks. Here's the full checklist. Run it on your own site today. The most common own-goal. Open your robots.txt it's at yoursite.com/robots.txt and look for lines like this: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / A lot of sites added these in 2023 when blocking AI training was the trend. The problem: the same bots that "train" also fetch pages to answer live questions. Block them and you opt out of being cited at all. Decide on purpose. If you want AI visibility, let them in: User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / Check your CDN too. Cloudflare and others ship a one-click "block AI bots" toggle that overrides your robots.txt . I've seen founders swear their site was open while Cloudflare quietly returned a 403 to every AI crawler. Test the real response, not the setting. This one kills more vibe-coded sites than anything else. Many crawlers fetch your raw HTML and don't run JavaScript, or run a limited version of it. If your page ships an empty <div id="root" and paints all the real text with client-side React, the crawler sees a blank page. Test it in ten seconds. Open your page, view source Ctrl+U , and search for a full sentence from your main content. If it's in the raw HTML, you're fine. If the source is just script tags and an empty shell, the crawler probably sees nothing. The fix is server-side rendering or static generation, so the words are in the HTML on first load. If you're on Next.js, Nuxt, Astro or plain HTML, you're likely fine. If you're on a pure client-side single-page app, this is your biggest leak. Crawlers and models favor pages that answer fast. If your real content sits below a hero section, a cookie banner, three testimonials and a newsletter box, the model has to dig for the passage worth quoting, and often it just moves to a cleaner source. Put a direct, quotable answer near the top. One clear paragraph that responds to the question the page targets, before any preamble. Write it so it can be lifted whole into an AI answer with nothing else attached. Models parse structure to understand and extract. Give them clear signals: ↳ One <h1 that states the topic, then <h2 s that each match a real follow-up question. ↳ Real lists and tables in HTML, not screenshots of lists. A model can quote a <table . It can't read a PNG. ↳ Short paragraphs. Dense walls of text are harder to extract a clean passage from. ↳ A visible last-updated date. Freshness is a trust signal, and models lean toward current pages. llms.txt is an emerging standard: a plain-text file at your root yoursite.com/llms.txt that hands AI systems a clean map of your most important pages, in Markdown, without the nav, ads and clutter of your full site. It's not yet honored by every model, and it won't rescue a page the crawler is otherwise blocked from. Treat it as cheap insurance, not a magic switch. A minimal one looks like this: Your Brand One sentence on what you do. Core pages - What is category https://yoursite.com/what-is : the definitive explainer - Your product vs alternatives https://yoursite.com/compare : honest comparison - Pricing https://yoursite.com/pricing : plans and what each includes Ten minutes to write, and it makes your best pages easy for a model to find and quote. There's little downside. A page nobody has crawled can't be cited. Check that your key pages are submitted in a sitemap, aren't marked noindex by accident, and return a clean 200, not a redirect chain or a soft 404. One stray noindex meta tag left over from a staging build can hide a page for months. Steps 1 and 2, the blocked-crawler and the JavaScript-shell problems, are the two that silently sink the most sites, and they're annoying to test properly across every bot. So we built a free tool that does it for you. Drop your URL into the AI crawler check at fulcru.app/tools/ai-crawler-check https://fulcru.app/tools/ai-crawler-check and it fetches your page the way GPTBot, ClaudeBot and PerplexityBot do, then tells you what each one actually sees: whether you're blocked, whether your content renders without JavaScript, and whether the answer is reachable. No card, no signup. It's the fastest way to catch the leak that's making all your good content invisible. Content strategy gets all the attention, but it sits on top of the plumbing. Fix the plumbing first: Let the AI crawlers in. Make sure your words are in the raw HTML. Answer in the first 100 words. Give the model clean structure to extract. Add an llms.txt as cheap insurance. Confirm your pages are indexed. Do this and every page you write afterward has a chance to be read. Skip it and the best AEO content in your category never gets seen by the one reader you're writing for. Run the crawler check today. It takes 30 seconds and it might explain a silence you've been blaming on your content. Want to see which questions ChatGPT, Gemini and Perplexity already answer with your competitor's name? Run the free Fulcru visibility report at fulcru.app.