{"slug": "is-your-site-invisible-to-chatgpt-check-your-robots-txt", "title": "Is your site invisible to ChatGPT? Check your robots.txt", "summary": "A developer at IndieDaddy published a free AI crawler checker that reports whether a site's robots.txt allows or blocks each major AI bot, including OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's Claude-SearchBot, Claude-User and ClaudeBot, PerplexityBot, Google-Extended, Bingbot and CCBot. The tool also flags whether an llms.txt file exists, and the writeup notes that starter templates and CDN \"block AI bots\" toggles often block search crawlers by default, keeping sites out of AI answers. The author recommends blocking training bots while allowing search bots, and keeping key product, audience and pricing details in plain HTML.", "body_md": "A lot of people now ask ChatGPT or Perplexity \"what's a good tool for X\" instead of Googling it. If your product never comes up, there might be a dull reason: your `robots.txt` is telling AI crawlers to stay out.\n\nSome starter templates do this. So do some CDN \"block AI bots\" toggles. You might not even know it's on.\n\nThis is the part most people miss. AI companies run separate crawlers for search and for training, so you can block one and allow the other.\n\n| Bot | Company | What it does | \n|---|---|---|\n| `OAI-SearchBot` | OpenAI | Shows your site in ChatGPT search answers | \n| `ChatGPT-User` | OpenAI | Opens a page when a user asks about it | \n| `GPTBot` | OpenAI | Collects pages for training | \n| `Claude-SearchBot` | Anthropic | Improves Claude's search results | \n| `Claude-User` | Anthropic | Opens a page when a user asks | \n| `ClaudeBot` | Anthropic | Collects pages for training | \n| `PerplexityBot` | Perplexity | Indexes sites for Perplexity answers | \n| `Google-Extended` |  | Gemini training and grounding, doesn't affect Google Search | \n| `Bingbot` | Microsoft | Bing, which Copilot also uses | \n| `CCBot` | Common Crawl | Open dataset a lot of models train on | \n\nIf the search ones (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) are blocked, assistants that search the web will struggle to cite you.\n\nOpen `https://yoursite.com/robots.txt` and look for something like:\n\n```\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nDisallow: /\n```\n\nA bot follows the group with its own name, and falls back to `*` if there isn't one. If two rules match, the longer path wins. `Disallow:` with nothing after it means everything is allowed, which trips people up.\n\nReading these by eye gets old fast, so I put a free [AI crawler checker](https://indiedaddy.com/free-tools/ai-crawler-checker) on IndieDaddy. It lists every bot above as allowed or blocked and tells you if you have an `llms.txt`. No signup, and it doesn't store anything.\n\nIf you want AI search but not training:\n\n```\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: *\nAllow: /\nDisallow: /api/\n\nSitemap: https://yoursite.com/sitemap.xml\n```\n\nFor most SaaS landing pages I'd just allow everything. The pages are public anyway, and getting cited is kind of the whole point.\n\n`llms.txt` is a proposed standard from 2024: a short Markdown file at `/llms.txt` that sums up your site for language models. Not every AI company reads it, so think of it as a small bonus.\n\n```\n# YourProduct\n\n> YourProduct helps freelance designers send invoices and get paid on time.\n> Pricing: free for 3 clients, Pro $12/month.\n\n## Key pages\n- Features: https://yourproduct.com/features\n- Pricing: https://yourproduct.com/pricing\n- Docs: https://yourproduct.com/docs\n\n## Not for\n- Payroll or accounting.\n```\n\nI'd keep the \"Not for\" bit. It stops assistants suggesting you for stuff you don't do.\n\nAssistants cite pages that are easy to understand and that say the same thing everywhere. Put what the product does, who it's for and the price in plain HTML, not only after a pile of JavaScript. Answer the question first on important pages, then go into detail. Use the same name and one-line description on your site, your directory listings and your socials.\n\nAnd get mentioned on sites the assistants already trust: decent directories, review sites, comparison posts. That part is slow and there's no shortcut.\n\nI wrote up the rest of it, including a 30-minute monthly check, in [how to get your SaaS recommended by ChatGPT, Claude and Perplexity](https://indiedaddy.com/blog/get-your-saas-recommended-by-chatgpt).\n\nDid you find any AI bots blocked on your own site? I was surprised how often it's on by default.", "url": "https://wpnews.pro/news/is-your-site-invisible-to-chatgpt-check-your-robots-txt", "canonical_source": "https://dev.to/indie_daddy/is-your-site-invisible-to-chatgpt-check-your-robotstxt-ojj", "published_at": "2026-11-04 20:19:02+00:00", "updated_at": "2026-11-04 20:23:57.488747+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "agent-protocols"], "entities": ["IndieDaddy", "OpenAI", "Anthropic", "Perplexity", "Google", "Microsoft", "Common Crawl", "ChatGPT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/is-your-site-invisible-to-chatgpt-check-your-robots-txt", "markdown": "https://wpnews.pro/news/is-your-site-invisible-to-chatgpt-check-your-robots-txt.md", "text": "https://wpnews.pro/news/is-your-site-invisible-to-chatgpt-check-your-robots-txt.txt", "jsonld": "https://wpnews.pro/news/is-your-site-invisible-to-chatgpt-check-your-robots-txt.jsonld"}}