# How to Make Your SaaS Docs and Product Pages Readable to AI Crawlers in 2027 > Source: > Published: 2026-11-05 04:42:46+00:00 Most AI crawlers read only the HTML your server sends on the first request, so a SaaS docs site or product page is readable to answer engines when its facts are in that HTML. In practice that means four things. Allow the search bots you want in robots.txt and in your firewall, render public pages on the server or at build time, keep navigation in plain anchor links, and add structured data that matches the visible text. Treat llms.txt as an optional extra, not a fix. The rest of this post shows how to test each of those and what to change first. This article was created with the help of AI and was researched, fact checked and edited by a human before publishing. A few quick definitions. AI visibility is how often AI engines such as ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews name and cite your product. GEO, or generative engine optimization, is the work of earning those mentions. AEO, or answer engine optimization, is shaping a page so an engine can lift a correct answer from it. None of that works if the crawler cannot read the page in the first place, and that part is an engineering problem. The ones that matter are the search and user fetch agents from OpenAI, Anthropic, Perplexity and Google, and each company now separates training from search. Here is what their own documentation says each agent does. | Agent | Company | What it does | What blocking it does | |---|---|---|---| | OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search features | Site is not shown in ChatGPT search answers, apart from navigational links | | GPTBot | OpenAI | Crawls content that may be used to train models | Signals your content should not be used for training | | ChatGPT-User | OpenAI | Visits pages when a user asks ChatGPT something | User initiated, so robots.txt rules may not apply | | Claude-SearchBot | Anthropic | Indexes content to improve Claude search results | May reduce visibility in Claude search answers | | ClaudeBot | Anthropic | Collects content that could contribute to training | Excludes future content from training datasets | | Claude-User | Anthropic | Fetches pages when a Claude user asks a question | May reduce visibility for user directed search | | PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search, not used for training | Removes you from Perplexity search results | | Perplexity-User | Perplexity | Fetches a page to answer a user question | Generally ignores robots.txt because a user asked | | Googlebot | | Crawls for Google Search, including AI Overviews and AI Mode | Removes you from Search and its AI features | | Google-Extended | | A control token for Gemini training and grounding in Gemini Apps and Vertex AI | Does not affect inclusion or ranking in Google Search | The useful pattern is that you can say no to training and still say yes to search. OpenAI states that each of its settings is independent, so you can allow OAI-SearchBot while disallowing GPTBot. Anthropic and Perplexity describe the same split. Write one group per agent and decide search access and training access separately. For most B2B SaaS companies the sensible default is to allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot on every public marketing and docs path, then make a deliberate policy call on GPTBot, ClaudeBot and Google-Extended. I am describing the rules in prose here rather than pasting a file, so treat this as a checklist for your own robots.txt. Expect a short delay. OpenAI says it can take about 24 hours for its systems to adjust after a robots.txt change, and Perplexity says up to 24 hours. One Google detail trips people up. Google-Extended does not control AI Overviews or AI Mode. Google says robots.txt rules for Googlebot are the control for AI features in Search, and that nosnippet, data-nosnippet, max-snippet and noindex limit what is shown. If you block Google-Extended to stay out of Gemini training, your AI Overviews eligibility does not change. Often it does not, and the firewall wins. A robots.txt that allows OAI-SearchBot means nothing if your CDN serves that agent a challenge page. In an October 2026 audit of 342 recently launched product sites, aat.ee found three that returned HTTP 403 to a request carrying the OAI-SearchBot user agent and HTTP 200 to the same request with a Chrome user agent. Two of them returned a Cloudflare challenge, and none of their robots.txt files disallowed AI crawlers. The audit requests came from the auditors' own network rather than OpenAI's IP ranges, so treat this as a prompt to check your settings, not proof that the real crawler is blocked. Perplexity's own docs add a sharper warning. Cloudflare's Bot Fight Mode cannot be skipped with WAF custom rules, so if it is on, it can still challenge Perplexity's bots even when you have written an allow rule. OpenAI and Perplexity both publish IP range files for their agents. Anthropic warns that IP blocking may not work reliably for its bots. The safest setup matches user agent and published IP range together, and you should check it after every security rule change. Mostly no, and that is the single biggest reason SaaS pages are invisible to answer engines. Google's stack renders JavaScript. The others largely read raw HTML. The best known evidence is a December 2024 study by Vercel and MERJ. It found that none of the major AI crawlers it examined rendered JavaScript, including OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot. ChatGPT's and Claude's crawlers did download JavaScript files, 11.50% and 23.84% of their requests, but did not execute them. Gemini, using Googlebot's infrastructure, did render. Newer tests point the same way. In a 41 day experiment published by Search Engine Land in August 2026, Vinicius Stanula split a site into sections with plain HTML links and sections whose links were injected by JavaScript. GPTBot and ClaudeBot each crawled all 748 HTML linked hierarchy pages and none of the JavaScript linked ones. Only Google's crawlers followed injected links, and Googlebot itself reached just 2% of the JavaScript linked pages. Real sites show the cost. In the aat.ee audit, 37 of 342 sites, or 10.8%, returned fewer than 50 words of body text in raw HTML. Thirty of those 37 served an empty root element and built the page in the browser, the default for a Vite or Create React App single page app. Only 1 of the 165 Next.js sites in the sample was thin. When the thin pages were loaded in a real browser, the median went from 0 words to about 475. The copy existed. The crawler just never received it. Fetch the page the way a non rendering crawler does and search the response for a sentence you care about. This takes two minutes per URL. ``` curl -sL -A "Mozilla/5.0 (compatible; OAI-SearchBot; +https://openai.com/searchbot)" -o raw.html -w "%{http_code}\n" https://example.com/pricing grep -c "a short phrase from your hero section" raw.html ``` A status other than 200 means the crawler gets an error or a challenge. A count of zero means the phrase is not in the HTML. Then check three more things. Fix rendering and links before anything else, because no other change matters if the crawler receives an empty shell. **Render public routes on the server or at build time.** Server side rendering, static generation or prerendering all put the copy in the first response. The logged in app can stay client rendered. Vercel's own recommendation is to server render main content, meta information and navigation. **Keep important copy out of components that fetch after mount.** In a Next.js App Router project, pages are server components by default, but a client component that loads feature text with an effect hook puts you back at square one for that block. **Use real anchor links for navigation.** Sidebar trees in docs frameworks are a common culprit when they are built from a client side config. If a crawler cannot see the link, it cannot find the child page. **Put tab and accordion content in the HTML.** Hidden by CSS is fine. Fetched on click is not. **Write code samples and limits as text.** Rate limits, supported languages, SDK names and plan limits belong in plain HTML, not screenshots. **Clean up dead URLs and redirects.** Vercel measured ChatGPT's crawler spending 34.82% of its fetches on 404 pages and Claude's 34.16%, mostly on outdated assets. Old versioned docs paths and stale static files waste the crawl. It helps engines label what is already on the page, but it is not a shortcut into answers. Google's guidance for AI Overviews and AI Mode is direct. There are no additional technical requirements, no special schema to add, and structured data should match the visible text. Correlation data is mild. In a Babel42 study of page one results published in October 2026, 35.1% of pages that AI Overviews ignored had no schema markup, against 28.5% of cited pages at the same organic position. That is a gap worth noticing, not a cause you can bank on. So ship Organization, SoftwareApplication or Product, and BreadcrumbList markup that says exactly what the page already says, and stop there. Markup cannot rescue a vague product description, it only tags it. It is cheap and harmless, but the evidence says AI crawlers barely read it. Ship it only after the basics above are done. An October 2026 roundup by AIVisibilityNerd compared 24 studies. Its own scan of the Tranco top 10,000 found a valid llms.txt on 9.7% of domains. On the usage side, Ahrefs reported that 97% of 38,360 customer sites with a valid file got no request for it in May 2026. Across 83 sites tracked by EZY, OpenAI's crawlers fetched llms.txt 7 times and robots.txt 3,990 times. Three separate citation studies found no measurable lift from having the file. Google adds that you do not need new machine readable files or AI text files to appear in its AI features. If you still publish one, keep it accurate. The same roundup found 2.2% of links inside sampled llms.txt files were dead, spread across 29.8% of the files that contained links. Watch verified crawler requests in your server logs, page by page, for a few weeks after you deploy. Filter by user agent and confirm the source IP against each company's published ranges, because user agent strings are easy to spoof. Recovery speed differs by crawler. In the Search Engine Land experiment, after the JavaScript links were converted to plain HTML, GPTBot swept the formerly hidden sections within two days and picked up 250 new pages. ClaudeBot kept recrawling pages it already knew and had not picked up a single newly linked page by day 41. A fix can land quickly for one engine and slowly for another, so log each one separately. Remember the limit of logs. A crawl is discovery, not indexing, and being readable does not mean being cited. I run an SEO and AI visibility practice called SearchAxe, and on every SaaS site we look at this is the first check, because it is a precondition for the rest. What other sources say about your product still decides most citations. That is a policy choice, not a visibility requirement. OpenAI says GPTBot and OAI-SearchBot are independent, so blocking GPTBot keeps content out of training while OAI-SearchBot can still surface you in ChatGPT search. No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are controlled through Googlebot rules and snippet controls. Usually not. Prerendering the public routes, such as home, pricing, features, integrations and docs, covers what assistants need. The logged in product can stay a client rendered app. Nobody can promise that. Readable HTML makes a citation possible. Whether one happens comes down to the quality of the answer on the page and the other trusted sources that mention you. If you have log data on how these crawlers treat your docs site, share what you found in the comments.