# Can ChatGPT and Claude read your website? A 5-minute checklist

> Source: <https://dev.to/pawe_iwanek_38733ca2d75c/can-chatgpt-and-claude-read-your-website-a-5-minute-checklist-5h94>
> Published: 2026-10-16 14:38:22+00:00

Search is changing: more people ask ChatGPT, Claude or Perplexity instead of opening Google. If those assistants cannot read your site, you will not be quoted. Here is a five-minute checklist.

Open `yoursite.com/robots.txt` and look for these user agents: `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended`, `CCBot`. A blanket `Disallow: /` under `User-agent: *` blocks all of them. Many CMS security plugins add it silently.

Run `curl -s https://yoursite.com | grep -i "your headline"`. If the headline is missing, your content is rendered by JavaScript, and most AI crawlers do not run JavaScript. They see an empty page.

A plain markdown file at `/llms.txt` that lists your key pages and describes the site. It is not a standard everyone honors yet, but it costs ten minutes and some assistants read it.

JSON-LD (`Organization`, `Product`, `Article`, `FAQPage`) tells a crawler what a page is without guessing. Validate it: a missing required property usually means the markup is ignored.

If your domain sends mail, check SPF, DKIM and DMARC. Without them a lot of your mail lands in spam, and Google and Yahoo now require them for bulk senders.

I got tired of doing this by hand, so I wrapped the checks into small tools on Apify. Each takes a domain and returns a 0-100 score with a concrete fix for every issue:

They are pay-per-result (0.005 USD per result), so scanning a handful of domains costs cents. Feedback and feature requests are welcome in the comments.
