A lot of people now ask ChatGPT or Perplexity "what's a good tool for X" instead of Googling it. If your product never comes up, there might be a dull reason: your robots.txt is telling AI crawlers to stay out.
Some starter templates do this. So do some CDN "block AI bots" toggles. You might not even know it's on.
This is the part most people miss. AI companies run separate crawlers for search and for training, so you can block one and allow the other.
| Bot | Company | What it does |
|---|---|---|
OAI-SearchBot |
OpenAI | Shows your site in ChatGPT search answers |
ChatGPT-User |
OpenAI | Opens a page when a user asks about it |
GPTBot |
OpenAI | Collects pages for training |
Claude-SearchBot |
Anthropic | Improves Claude's search results |
Claude-User |
Anthropic | Opens a page when a user asks |
ClaudeBot |
Anthropic | Collects pages for training |
PerplexityBot |
Perplexity | Indexes sites for Perplexity answers |
Google-Extended |
Gemini training and grounding, doesn't affect Google Search | |
Bingbot |
Microsoft | Bing, which Copilot also uses |
CCBot |
Common Crawl | Open dataset a lot of models train on |
If the search ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are blocked, assistants that search the web will struggle to cite you.
Open https://yoursite.com/robots.txt and look for something like:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /
A bot follows the group with its own name, and falls back to * if there isn't one. If two rules match, the longer path wins. Disallow: with nothing after it means everything is allowed, which trips people up.
Reading these by eye gets old fast, so I put a free AI crawler checker on IndieDaddy. It lists every bot above as allowed or blocked and tells you if you have an llms.txt. No signup, and it doesn't store anything.
If you want AI search but not training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://yoursite.com/sitemap.xml
For most SaaS landing pages I'd just allow everything. The pages are public anyway, and getting cited is kind of the whole point.
llms.txt is a proposed standard from 2024: a short Markdown file at /llms.txt that sums up your site for language models. Not every AI company reads it, so think of it as a small bonus.
> YourProduct helps freelance designers send invoices and get paid on time.
> Pricing: free for 3 clients, Pro $12/month.
## Key pages
- Features: https://yourproduct.com/features
- Pricing: https://yourproduct.com/pricing
- Docs: https://yourproduct.com/docs
## Not for
- Payroll or accounting.
I'd keep the "Not for" bit. It stops assistants suggesting you for stuff you don't do.
Assistants cite pages that are easy to understand and that say the same thing everywhere. Put what the product does, who it's for and the price in plain HTML, not only after a pile of JavaScript. Answer the question first on important pages, then go into detail. Use the same name and one-line description on your site, your directory listings and your socials.
And get mentioned on sites the assistants already trust: decent directories, review sites, comparison posts. That part is slow and there's no shortcut.
I wrote up the rest of it, including a 30-minute monthly check, in how to get your SaaS recommended by ChatGPT, Claude and Perplexity.
Did you find any AI bots blocked on your own site? I was surprised how often it's on by default.