# How to check whether AI crawlers can read your site (robots.txt and server logs) > Source: > Published: 2026-10-22 20:56:36+00:00 AI answer engines only recommend or cite what they can fetch. Before you invest in GEO, confirm the crawlers can actually read your site. Here is a quick check. Note that OAI-SearchBot and ChatGPT-User also exist for search and live browsing, so check them if you want ChatGPT to cite you. Open `https://yourdomain.com/robots.txt`. A blanket `Disallow: /` under `User-agent: *`, or a rule copied from a security plugin, often blocks AI bots by accident. To explicitly allow them: ``` User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://yourdomain.com/sitemap.xml ``` If you prefer to keep some areas private, add `Disallow: /private/` inside the relevant block. robots.txt is a request, not enforcement, and CDNs or firewalls (Cloudflare bot rules, WAFs) can block bots before they ever see it. Your access log shows what really happens: ``` grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | tail -n 20 ``` Count hits per bot and status code: ``` grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c ``` Look at the status codes. 200 means the bot read the page. 403 or 429 means your firewall or rate limiter is rejecting it. No hits at all over a couple of weeks may mean it is blocked upstream, or simply that your site is not yet discovered. Note that Google-Extended will not appear in logs, since Googlebot does the fetching. Whitelist the bots in your CDN or WAF, keep pages server-rendered (many AI crawlers do not run JavaScript), and make sure your sitemap lists the key pages. Then re-check the logs after a week. We wrote a longer glossary of overseas-ad and AI-search terms here: [https://www.pandawm.com/blog/overseas-advertising-glossary.html](https://www.pandawm.com/blog/overseas-advertising-glossary.html)