{"slug": "how-to-check-whether-ai-crawlers-can-read-your-site-robots-txt-and-server-logs", "title": "How to check whether AI crawlers can read your site (robots.txt and server logs)", "summary": "A developer published a step-by-step method for verifying whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot can actually fetch a site, starting with the robots.txt file and confirming real behavior through nginx access logs. The guide recommends explicit Allow rules for each bot, grepping logs for hits and HTTP status codes (200 vs. 403/429), and whitelisting crawlers in CDNs or WAFs, since robots.txt is only a request and firewalls can block bots before it is read. It also notes that pages should be server-rendered because many AI crawlers do not execute JavaScript.", "body_md": "AI answer engines only recommend or cite what they can fetch. Before you invest in GEO, confirm the crawlers can actually read your site. Here is a quick check.\n\nNote that OAI-SearchBot and ChatGPT-User also exist for search and live browsing, so check them if you want ChatGPT to cite you.\n\nOpen `https://yourdomain.com/robots.txt`. A blanket `Disallow: /` under `User-agent: *`, or a rule copied from a security plugin, often blocks AI bots by accident. To explicitly allow them:\n\n```\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /\n\nSitemap: https://yourdomain.com/sitemap.xml\n```\n\nIf you prefer to keep some areas private, add `Disallow: /private/` inside the relevant block.\n\nrobots.txt is a request, not enforcement, and CDNs or firewalls (Cloudflare bot rules, WAFs) can block bots before they ever see it. Your access log shows what really happens:\n\n```\ngrep -Ei \"GPTBot|ClaudeBot|PerplexityBot\" /var/log/nginx/access.log | tail -n 20\n```\n\nCount hits per bot and status code:\n\n```\ngrep -Ei \"GPTBot|ClaudeBot|PerplexityBot\" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c\n```\n\nLook at the status codes. 200 means the bot read the page. 403 or 429 means your firewall or rate limiter is rejecting it. No hits at all over a couple of weeks may mean it is blocked upstream, or simply that your site is not yet discovered. Note that Google-Extended will not appear in logs, since Googlebot does the fetching.\n\nWhitelist the bots in your CDN or WAF, keep pages server-rendered (many AI crawlers do not run JavaScript), and make sure your sitemap lists the key pages. Then re-check the logs after a week.\n\nWe wrote a longer glossary of overseas-ad and AI-search terms here: [https://www.pandawm.com/blog/overseas-advertising-glossary.html](https://www.pandawm.com/blog/overseas-advertising-glossary.html)", "url": "https://wpnews.pro/news/how-to-check-whether-ai-crawlers-can-read-your-site-robots-txt-and-server-logs", "canonical_source": "https://dev.to/pandawm/how-to-check-whether-ai-crawlers-can-read-your-site-robotstxt-and-server-logs-3c86", "published_at": "2026-10-22 20:56:36+00:00", "updated_at": "2026-10-22 21:10:51.254915+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search"], "entities": ["GPTBot", "ClaudeBot", "PerplexityBot", "Google-Extended", "OAI-SearchBot", "ChatGPT-User", "Cloudflare"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-check-whether-ai-crawlers-can-read-your-site-robots-txt-and-server-logs", "markdown": "https://wpnews.pro/news/how-to-check-whether-ai-crawlers-can-read-your-site-robots-txt-and-server-logs.md", "text": "https://wpnews.pro/news/how-to-check-whether-ai-crawlers-can-read-your-site-robots-txt-and-server-logs.txt", "jsonld": "https://wpnews.pro/news/how-to-check-whether-ai-crawlers-can-read-your-site-robots-txt-and-server-logs.jsonld"}}