Can AI crawlers read your site? robots.txt, llms.txt and JavaScript rendering, checked

Raphaël Aubry, founder of Howseen, published a guide showing how to check from the terminal whether AI crawlers can read a site, covering robots.txt rules, llms.txt, and JavaScript rendering. The guide notes that OpenAI, Anthropic, and Perplexity each run separate crawlers for training, search indexing, and user-triggered fetches, and that most AI crawlers parse raw HTML without executing JavaScript, so client-rendered pages return empty shells. Aubry also released a free tool that runs the same checks across up to 100 URLs.

Originally published on howseen.ai: howseen.ai/llms-txt https://howseen.ai/llms-txt?utm source=devto&utm medium=referral&utm campaign=o2-syndication By Raphaël Aubry, founder of Howseen An AI assistant can only cite a page it can read. Before anyone talks about "GEO" or content strategy, three boring things decide whether your site is even in the running: robots.txt let the AI crawlers in? llms.txt , and does it matter? Here is how to check all three from a terminal, what the official documentation says, and a free tool that runs the same checks on up to 100 URLs at once. curl -s https://yoursite.com/robots.txt The thing most people miss: AI companies run several crawlers with different jobs, and each one reads its own group in robots.txt. From each company's documentation: | Company | Token | Job | |---|---|---| | OpenAI | GPTBot | Collects content for model training | | OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search | | OpenAI | ChatGPT-User | Fetches pages for user actions in ChatGPT | | Anthropic | ClaudeBot | Collects content for model training | | Anthropic | Claude-SearchBot | Indexes content for Claude search results | | Anthropic | Claude-User | Fetches pages when a Claude user asks | | Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search | | Perplexity | Perplexity-User | Fetches pages when a user asks a question | | | Google-Extended | Token for Gemini training and grounding, not a separate crawler | Three consequences: GPTBot training and keep OAI-SearchBot ChatGPT search . OpenAI says sites that block OAI-SearchBot do not appear in ChatGPT search answers. Google-Extended does not control AI Overviews. User-agent: followed by Disallow: / blocks every bot that has no group of its own, AI crawlers included. In my experience it is usually a leftover from an old scraper defense that nobody reopened. Reading robots.txt by eye works until rules overlap. The standard RFC 9309 https://www.rfc-editor.org/rfc/rfc9309 says the most specific user-agent group wins, then the longest matching rule, and Allow wins a tie. Here is that logic in the checker I describe at the end of this post: js const specific = parsed.groups.filter g = g.agents.includes token ; if specific.length { matchedGroup = token; rules = specific.flatMap g = g.rules ; } else { const star = parsed.groups.filter g = g.agents.includes ' ' ; if star.length { matchedGroup = ' '; rules = star.flatMap g = g.rules ; } } let best = null; for const r of rules { if ruleMatches r.path, path continue; const len = r.path.length; if best || len best.path.length || len === best.path.length && r.type === 'allow' best = r; } const allowed = best || best.type === 'allow'; Note the fallback: a bot with its own group ignores the group entirely. Adding User-agent: GPTBot with a single Disallow: /private/ means GPTBot no longer sees anything you wrote under . One more layer sits before robots.txt: your firewall or CDN. A WAF rule that blocks unknown bots can reject AI crawlers before they read a single line. If your logs show no hits from these tokens at all, check there too. Most AI crawlers fetch the raw HTML and parse it. They do not run your JavaScript. If your page is a client-rendered shell, they get an empty <div id="root" . Fetch the page with a crawler user agent and count the words that are actually in the HTML: UA="Mozilla/5.0 AppleWebKit/537.36 KHTML, like Gecko ; compatible; GPTBot/1.4; +https://openai.com/gptbot" curl -s -A "$UA" https://yoursite.com/pricing \ | perl -0pe 's/<script. ?<\/script //gs; s/<style. ?<\/style //gs; s/< ^ + / /g' \ | wc -w It is crude navigation and footer count too , but the order of magnitude tells the story. A few dozen words on a page that looks full in the browser means the content arrives through JavaScript. The same command also shows whether your server or CDN answers this user agent with a 200 at all: add -o /dev/null -w "%{http code}\n" . The checker uses the same idea with explicit thresholds: under 100 words, the page scores 0 on this check, and if the HTML also contains a framework root, it is flagged as rendered in the browser. js const SPA MARKERS = 'id="root"', 'React' , 'id=" next"', 'Next.js' , 'id="app"', 'Vue or similar' , 'id=" nuxt"', 'Nuxt' , 'ng-version', 'Angular' , '<app-root', 'Angular' , 'id="svelte"', 'Svelte' , 'data-reactroot', 'React' , ; const likelyClientRendered = wordCount < 100 && markers.length 0 || /<script\b ^ src=/i.test html ; The fix depends on the stack: server components in the Next.js App Router, SSR mode in Nuxt, pre-rendering or static generation for a React SPA. You can also do the manual test: View Page Source not Inspect and look for your headings. llms.txt is a Markdown file at the root of your site that summarises it and links the key pages. Jeremy Howard proposed the format on September 3, 2024, at llmstxt.org. The only required part is an H1 with the site name. curl -s -o /dev/null -w "%{http code} %{content type}\n" https://yoursite.com/llms.txt 200 text/plain; charset=utf-8 <- what you want The classic failure is a site that answers every unknown path with its home page: you get a 200, but with text/html . The checker treats that as "no llms.txt". Now the honest part. No major AI engine has confirmed using llms.txt to decide which brands to cite. Google says you don't need to create new machine readable files, AI text files, or markup to appear in AI Overviews and AI Mode. OpenAI points site owners to robots.txt and OAI-SearchBot for ChatGPT search. Where llms.txt is used is inference on documentation: OpenAI publishes its own docs index as an llms.txt file, and AI coding tools read these files to land on the right page. | File | What it does | Effect on AI visibility | |---|---|---| | robots.txt | Allows or blocks crawlers | Direct: a blocked crawler cannot cite you | | sitemap.xml | Lists the URLs you want indexed | Indirect: helps pages get indexed | | llms.txt | Summarises the site, links key pages | None confirmed by a major AI engine | So: write one if you have docs, an API or many pages it takes minutes , and check robots.txt first. I packaged these checks as a free Apify Actor, AI Visibility Checker https://apify.com/elraphh/ai-visibility-checker . Plain HTTP requests, no AI calls, no third party login. You paste up to 100 URLs per run and get a score out of 100 with one sentence per fix: | Check | Points | |---|---| | AI search and assistant crawlers in robots.txt | 25 | | Text readable without JavaScript | 20 | | Structured data JSON-LD | 15 | | XML sitemap | 7 | | AI training crawlers in robots.txt | 5 | | Indexable, title, meta description, canonical, llms.txt | 5 each | | <html lang | 3 | Training crawlers weigh little on purpose: blocking them is a valid choice. robots.txt, llms.txt and the sitemap are fetched once per domain, even if you check many pages of the same site. { "urls": "https://example.com", "https://example.com/pricing" , "timeoutSecs": 15 } A useful pattern is to schedule it after each deploy, so a robots.txt change or a rendering regression shows up the same day instead of months later. For the per-bot details user agent strings, published IP ranges, how to spot a fake GPTBot in your logs , see the AI crawlers robots.txt guide https://howseen.ai/blog/ai-crawlers-explained-gptbot-claudebot-perplexitybot-robots-txt?utm source=devto&utm medium=referral&utm campaign=o2-syndication . See what AI says about your brand: howseen.ai https://howseen.ai/?utm source=devto&utm medium=referral&utm campaign=o2-syndication