AI answer engines only recommend or cite what they can fetch. Before you invest in GEO, confirm the crawlers can actually read your site. Here is a quick check.
Note that OAI-SearchBot and ChatGPT-User also exist for search and live browsing, so check them if you want ChatGPT to cite you.
Open https://yourdomain.com/robots.txt. A blanket Disallow: / under User-agent: *, or a rule copied from a security plugin, often blocks AI bots by accident. To explicitly allow them:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
If you prefer to keep some areas private, add Disallow: /private/ inside the relevant block.
robots.txt is a request, not enforcement, and CDNs or firewalls (Cloudflare bot rules, WAFs) can block bots before they ever see it. Your access log shows what really happens:
grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | tail -n 20
Count hits per bot and status code:
grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
Look at the status codes. 200 means the bot read the page. 403 or 429 means your firewall or rate limiter is rejecting it. No hits at all over a couple of weeks may mean it is blocked upstream, or simply that your site is not yet discovered. Note that Google-Extended will not appear in logs, since Googlebot does the fetching.
Whitelist the bots in your CDN or WAF, keep pages server-rendered (many AI crawlers do not run JavaScript), and make sure your sitemap lists the key pages. Then re-check the logs after a week.
We wrote a longer glossary of overseas-ad and AI-search terms here: https://www.pandawm.com/blog/overseas-advertising-glossary.html