Can AI crawlers read your site? robots.txt, llms.txt and JavaScript rendering, checked

Originally published on howseen.ai: howseen.ai/llms-txt

By Raphaël Aubry, founder of Howseen

An AI assistant can only cite a page it can read. Before anyone talks about "GEO" or content strategy, three boring things decide whether your site is even in the running:

robots.txt let the AI crawlers in?llms.txt, and does it matter? Here is how to check all three from a terminal, what the official documentation says, and a free tool that runs the same checks on up to 100 URLs at once.

curl -s https://yoursite.com/robots.txt

The thing most people miss: AI companies run several crawlers with different jobs, and each one reads its own group in robots.txt. From each company's documentation:

Three consequences:

GPTBot (training) and keep OAI-SearchBot (ChatGPT search). OpenAI says sites that block OAI-SearchBot do not appear in ChatGPT search answers.Google-Extended does not control AI Overviews.User-agent: * followed by Disallow: / blocks every bot that has no group of its own, AI crawlers included. In my experience it is usually a leftover from an old scraper defense that nobody reopened. Reading robots.txt by eye works until rules overlap. The standard (RFC 9309) says the most specific user-agent group wins, then the longest matching rule, and Allow wins a tie. Here is that logic in the checker I describe at the end of this post:

const specific = parsed.groups.filter((g) => g.agents.includes(token));
if (specific.length) {
    matchedGroup = token;
    rules = specific.flatMap((g) => g.rules);
} else {
    const star = parsed.groups.filter((g) => g.agents.includes('*'));
    if (star.length) {
        matchedGroup = '*';
        rules = star.flatMap((g) => g.rules);
    }
}
let best = null;
for (const r of rules) {
    if (!ruleMatches(r.path, path)) continue;
    const len = r.path.length;
    if (!best || len > best.path.length || (len === best.path.length && r.type === 'allow')) best = r;
}
const allowed = !best || best.type === 'allow';

Note the fallback: a bot with its own group ignores the * group entirely. Adding User-agent: GPTBot with a single Disallow: /private/ means GPTBot no longer sees anything you wrote under *.

One more layer sits before robots.txt: your firewall or CDN. A WAF rule that blocks unknown bots can reject AI crawlers before they read a single line. If your logs show no hits from these tokens at all, check there too.

Most AI crawlers fetch the raw HTML and parse it. They do not run your JavaScript. If your page is a client-rendered shell, they get an empty <div id="root">.

Fetch the page with a crawler user agent and count the words that are actually in the HTML:

UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

curl -s -A "$UA" https://yoursite.com/pricing \
  | perl -0pe 's/<script.*?<\/script>//gs; s/<style.*?<\/style>//gs; s/<[^>]+>/ /g' \
  | wc -w

It is crude (navigation and footer count too), but the order of magnitude tells the story. A few dozen words on a page that looks full in the browser means the content arrives through JavaScript. The same command also shows whether your server or CDN answers this user agent with a 200 at all: add -o /dev/null -w "%{http_code}\n".

The checker uses the same idea with explicit thresholds: under 100 words, the page scores 0 on this check, and if the HTML also contains a framework root, it is flagged as rendered in the browser.

const SPA_MARKERS = [
    ['id="root"', 'React'],
    ['id="__next"', 'Next.js'],
    ['id="app"', 'Vue or similar'],
    ['id="__nuxt"', 'Nuxt'],
    ['ng-version', 'Angular'],
    ['<app-root', 'Angular'],
    ['id="svelte"', 'Svelte'],
    ['data-reactroot', 'React'],
];
const likelyClientRendered = wordCount < 100 && (markers.length > 0 || /<script\b[^>]*src=/i.test(html));

The fix depends on the stack: server components in the Next.js App Router, SSR mode in Nuxt, pre-rendering or static generation for a React SPA. You can also do the manual test: View Page Source (not Inspect) and look for your headings.

llms.txt is a Markdown file at the root of your site that summarises it and links the key pages. Jeremy Howard proposed the format on September 3, 2024, at llmstxt.org. The only required part is an H1 with the site name.

curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://yoursite.com/llms.txt

The classic failure is a site that answers every unknown path with its home page: you get a 200, but with text/html. The checker treats that as "no llms.txt".

Now the honest part. No major AI engine has confirmed using llms.txt to decide which brands to cite. Google says you don't need to create new machine readable files, AI text files, or markup to appear in AI Overviews and AI Mode. OpenAI points site owners to robots.txt and OAI-SearchBot for ChatGPT search. Where llms.txt is used is inference on documentation: OpenAI publishes its own docs index as an llms.txt file, and AI coding tools read these files to land on the right page.

So: write one if you have docs, an API or many pages (it takes minutes), and check robots.txt first.

I packaged these checks as a free Apify Actor, AI Visibility Checker. Plain HTTP requests, no AI calls, no third party login. You paste up to 100 URLs per run and get a score out of 100 with one sentence per fix:

Training crawlers weigh little on purpose: blocking them is a valid choice. robots.txt, llms.txt and the sitemap are fetched once per domain, even if you check many pages of the same site.

{
  "urls": ["https://example.com", "https://example.com/pricing"],
  "timeoutSecs": 15
}

A useful pattern is to schedule it after each deploy, so a robots.txt change or a rendering regression shows up the same day instead of months later.

For the per-bot details (user agent strings, published IP ranges, how to spot a fake GPTBot in your logs), see the AI crawlers robots.txt guide.

See what AI says about your brand: howseen.ai