Is your site invisible to ChatGPT? Check your robots.txt

A developer at IndieDaddy published a free AI crawler checker that reports whether a site's robots.txt allows or blocks each major AI bot, including OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's Claude-SearchBot, Claude-User and ClaudeBot, PerplexityBot, Google-Extended, Bingbot and CCBot. The tool also flags whether an llms.txt file exists, and the writeup notes that starter templates and CDN "block AI bots" toggles often block search crawlers by default, keeping sites out of AI answers. The author recommends blocking training bots while allowing search bots, and keeping key product, audience and pricing details in plain HTML.

A lot of people now ask ChatGPT or Perplexity "what's a good tool for X" instead of Googling it. If your product never comes up, there might be a dull reason: your robots.txt is telling AI crawlers to stay out. Some starter templates do this. So do some CDN "block AI bots" toggles. You might not even know it's on. This is the part most people miss. AI companies run separate crawlers for search and for training, so you can block one and allow the other. | Bot | Company | What it does | |---|---|---| | OAI-SearchBot | OpenAI | Shows your site in ChatGPT search answers | | ChatGPT-User | OpenAI | Opens a page when a user asks about it | | GPTBot | OpenAI | Collects pages for training | | Claude-SearchBot | Anthropic | Improves Claude's search results | | Claude-User | Anthropic | Opens a page when a user asks | | ClaudeBot | Anthropic | Collects pages for training | | PerplexityBot | Perplexity | Indexes sites for Perplexity answers | | Google-Extended | | Gemini training and grounding, doesn't affect Google Search | | Bingbot | Microsoft | Bing, which Copilot also uses | | CCBot | Common Crawl | Open dataset a lot of models train on | If the search ones OAI-SearchBot , Claude-SearchBot , PerplexityBot are blocked, assistants that search the web will struggle to cite you. Open https://yoursite.com/robots.txt and look for something like: User-agent: GPTBot Disallow: / User-agent: Disallow: / A bot follows the group with its own name, and falls back to if there isn't one. If two rules match, the longer path wins. Disallow: with nothing after it means everything is allowed, which trips people up. Reading these by eye gets old fast, so I put a free AI crawler checker https://indiedaddy.com/free-tools/ai-crawler-checker on IndieDaddy. It lists every bot above as allowed or blocked and tells you if you have an llms.txt . No signup, and it doesn't store anything. If you want AI search but not training: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Allow: / Disallow: /api/ Sitemap: https://yoursite.com/sitemap.xml For most SaaS landing pages I'd just allow everything. The pages are public anyway, and getting cited is kind of the whole point. llms.txt is a proposed standard from 2024: a short Markdown file at /llms.txt that sums up your site for language models. Not every AI company reads it, so think of it as a small bonus. YourProduct YourProduct helps freelance designers send invoices and get paid on time. Pricing: free for 3 clients, Pro $12/month. Key pages - Features: https://yourproduct.com/features - Pricing: https://yourproduct.com/pricing - Docs: https://yourproduct.com/docs Not for - Payroll or accounting. I'd keep the "Not for" bit. It stops assistants suggesting you for stuff you don't do. Assistants cite pages that are easy to understand and that say the same thing everywhere. Put what the product does, who it's for and the price in plain HTML, not only after a pile of JavaScript. Answer the question first on important pages, then go into detail. Use the same name and one-line description on your site, your directory listings and your socials. And get mentioned on sites the assistants already trust: decent directories, review sites, comparison posts. That part is slow and there's no shortcut. I wrote up the rest of it, including a 30-minute monthly check, in how to get your SaaS recommended by ChatGPT, Claude and Perplexity https://indiedaddy.com/blog/get-your-saas-recommended-by-chatgpt . Did you find any AI bots blocked on your own site? I was surprised how often it's on by default.