{"slug": "can-chatgpt-and-claude-read-your-website-a-5-minute-checklist", "title": "Can ChatGPT and Claude read your website? A 5-minute checklist", "summary": "A developer published a five-minute checklist for making websites readable by AI assistants such as ChatGPT, Claude and Perplexity, covering robots.txt user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot), JavaScript-rendered content that AI crawlers cannot execute, an /llms.txt file, JSON-LD structured data, and SPF/DKIM/DMARC email authentication. The author wrapped the checks into pay-per-result tools on Apify that score a domain from 0 to 100 and return a concrete fix for each issue, priced at 0.005 USD per result.", "body_md": "Search is changing: more people ask ChatGPT, Claude or Perplexity instead of opening Google. If those assistants cannot read your site, you will not be quoted. Here is a five-minute checklist.\n\nOpen `yoursite.com/robots.txt` and look for these user agents: `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended`, `CCBot`. A blanket `Disallow: /` under `User-agent: *` blocks all of them. Many CMS security plugins add it silently.\n\nRun `curl -s https://yoursite.com | grep -i \"your headline\"`. If the headline is missing, your content is rendered by JavaScript, and most AI crawlers do not run JavaScript. They see an empty page.\n\nA plain markdown file at `/llms.txt` that lists your key pages and describes the site. It is not a standard everyone honors yet, but it costs ten minutes and some assistants read it.\n\nJSON-LD (`Organization`, `Product`, `Article`, `FAQPage`) tells a crawler what a page is without guessing. Validate it: a missing required property usually means the markup is ignored.\n\nIf your domain sends mail, check SPF, DKIM and DMARC. Without them a lot of your mail lands in spam, and Google and Yahoo now require them for bulk senders.\n\nI got tired of doing this by hand, so I wrapped the checks into small tools on Apify. Each takes a domain and returns a 0-100 score with a concrete fix for every issue:\n\nThey are pay-per-result (0.005 USD per result), so scanning a handful of domains costs cents. Feedback and feature requests are welcome in the comments.", "url": "https://wpnews.pro/news/can-chatgpt-and-claude-read-your-website-a-5-minute-checklist", "canonical_source": "https://dev.to/pawe_iwanek_38733ca2d75c/can-chatgpt-and-claude-read-your-website-a-5-minute-checklist-5h94", "published_at": "2026-10-16 14:38:22+00:00", "updated_at": "2026-10-16 14:47:28.205605+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "structured-data", "agent-protocols"], "entities": ["ChatGPT", "Claude", "Perplexity", "GPTBot", "ClaudeBot", "PerplexityBot", "Google", "Apify"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/can-chatgpt-and-claude-read-your-website-a-5-minute-checklist", "markdown": "https://wpnews.pro/news/can-chatgpt-and-claude-read-your-website-a-5-minute-checklist.md", "text": "https://wpnews.pro/news/can-chatgpt-and-claude-read-your-website-a-5-minute-checklist.txt", "jsonld": "https://wpnews.pro/news/can-chatgpt-and-claude-read-your-website-a-5-minute-checklist.jsonld"}}