{"slug": "three-quick-checks-before-you-expect-ai-search-to-quote-your-page", "title": "Three quick checks before you expect AI search to quote your page", "summary": "A developer built small Claude/Cursor skills that audit web pages for AI-search readiness and found three recurring gaps: missing or contradictory AI-crawler rules in robots.txt, absent or stale llms.txt files, and JSON-LD structured data that doesn't match the visible page. Checking public sites, the developer found stripe.com served an llms.txt while anthropic.com returned 404, and one payments explainer had a null author, missing dateModified and a brand-suffixed headline. The developer cautions the checks only remove cheap reasons a page might be skipped, not guarantee citation.", "body_md": "I've been building small Claude/Cursor skills that review pages for AI-search readiness. Running them on public pages kept turning up the same three gaps. You can check all three yourself in about ten minutes with curl and view-source. You don't need a tool.\n\n```\ncurl -s https://yoursite.com/robots.txt | grep -iE 'gptbot|claudebot|perplexitybot|google-extended|ccbot'\n```\n\nNo output means you have no AI-specific rules, so these bots follow your `User-agent: *` group. That's fine if it's on purpose. The problems I see are leftovers: a blanket `Disallow` someone added years ago, or a CDN/WAF bot rule that blocks crawlers the robots.txt says are allowed. Check both.\n\n```\ncurl -s -o /dev/null -w '%{http_code}\\n' https://yoursite.com/llms.txt\n```\n\nllms.txt is a proposed convention, not a standard, and no AI company has promised to read it. It's still cheap to add. Put a one-line summary of what you do at the top, then short sections linking your key pages. When I checked on Nov 13, stripe.com returned 200 with exactly that layout, and anthropic.com returned 404. If you add one, keep it current. A stale llms.txt is worse than none.\n\nOpen view-source, search for `application/ld+json`, and compare it with what a reader sees:\n\nOn one well-known payments explainer, all three were off: author was null, dateModified was missing even though the page showed a date, and the headline had a brand suffix. These take minutes to fix. Don't add FAQ or rating markup the page doesn't actually show.\n\nNone of this guarantees a citation. Nobody outside the AI companies knows exactly how sources get chosen. These checks only remove the cheap reasons a page might be skipped or misread. Whether people are asking the question your page answers, and whether you answer it well in the first paragraph, matters more.\n\n*Disclosure: I make HubDev, a set of Claude/Cursor skills that run checks like these. If you'd rather have an agent score a page for you, there's a free starter scorecard (no purchase needed): [https://payhip.com/b/TvxJH](https://payhip.com/b/TvxJH). Paid packs exist, but nothing in this post needs them.*", "url": "https://wpnews.pro/news/three-quick-checks-before-you-expect-ai-search-to-quote-your-page", "canonical_source": "https://dev.to/hubdev/three-quick-checks-before-you-expect-ai-search-to-quote-your-page-1le2", "published_at": "2026-11-13 20:14:35+00:00", "updated_at": "2026-11-13 20:21:01.550243+00:00", "lang": "en", "topics": ["generative-engine-optimization", "ai-crawlers", "ai-search", "structured-data", "ai-tools"], "entities": ["Claude", "Cursor", "Stripe", "Anthropic", "HubDev"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/three-quick-checks-before-you-expect-ai-search-to-quote-your-page", "markdown": "https://wpnews.pro/news/three-quick-checks-before-you-expect-ai-search-to-quote-your-page.md", "text": "https://wpnews.pro/news/three-quick-checks-before-you-expect-ai-search-to-quote-your-page.txt", "jsonld": "https://wpnews.pro/news/three-quick-checks-before-you-expect-ai-search-to-quote-your-page.jsonld"}}