{"slug": "can-ai-crawlers-read-your-site-robots-txt-llms-txt-and-javascript-rendering", "title": "Can AI crawlers read your site? robots.txt, llms.txt and JavaScript rendering, checked", "summary": "Raphaël Aubry, founder of Howseen, published a guide showing how to check from the terminal whether AI crawlers can read a site, covering robots.txt rules, llms.txt, and JavaScript rendering. The guide notes that OpenAI, Anthropic, and Perplexity each run separate crawlers for training, search indexing, and user-triggered fetches, and that most AI crawlers parse raw HTML without executing JavaScript, so client-rendered pages return empty shells. Aubry also released a free tool that runs the same checks across up to 100 URLs.", "body_md": "*Originally published on howseen.ai: [howseen.ai/llms-txt](https://howseen.ai/llms-txt?utm_source=devto&utm_medium=referral&utm_campaign=o2-syndication)*\n\n*By Raphaël Aubry, founder of Howseen*\n\nAn AI assistant can only cite a page it can read. Before anyone talks about \"GEO\" or content strategy, three boring things decide whether your site is even in the running:\n\n`robots.txt` let the AI crawlers in?`llms.txt`, and does it matter?\nHere is how to check all three from a terminal, what the official documentation says, and a free tool that runs the same checks on up to 100 URLs at once.\n\n```\ncurl -s https://yoursite.com/robots.txt\n```\n\nThe thing most people miss: AI companies run several crawlers with different jobs, and each one reads its own group in robots.txt. From each company's documentation:\n\n| Company | Token | Job | \n|---|---|---|\n| OpenAI | `GPTBot` | Collects content for model training | \n| OpenAI | `OAI-SearchBot` | Surfaces sites in ChatGPT search | \n| OpenAI | `ChatGPT-User` | Fetches pages for user actions in ChatGPT | \n| Anthropic | `ClaudeBot` | Collects content for model training | \n| Anthropic | `Claude-SearchBot` | Indexes content for Claude search results | \n| Anthropic | `Claude-User` | Fetches pages when a Claude user asks | \n| Perplexity | `PerplexityBot` | Surfaces and links sites in Perplexity search | \n| Perplexity | `Perplexity-User` | Fetches pages when a user asks a question | \n|  | `Google-Extended` | Token for Gemini training and grounding, not a separate crawler | \n\nThree consequences:\n\n`GPTBot` (training) and keep `OAI-SearchBot` (ChatGPT search). OpenAI says sites that block OAI-SearchBot do not appear in ChatGPT search answers.`Google-Extended` does not control AI Overviews.`User-agent: *` followed by `Disallow: /` blocks every bot that has no group of its own, AI crawlers included. In my experience it is usually a leftover from an old scraper defense that nobody reopened.\nReading robots.txt by eye works until rules overlap. The standard ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)) says the most specific user-agent group wins, then the longest matching rule, and `Allow` wins a tie. Here is that logic in the checker I describe at the end of this post:\n\n``` js\nconst specific = parsed.groups.filter((g) => g.agents.includes(token));\nif (specific.length) {\n    matchedGroup = token;\n    rules = specific.flatMap((g) => g.rules);\n} else {\n    const star = parsed.groups.filter((g) => g.agents.includes('*'));\n    if (star.length) {\n        matchedGroup = '*';\n        rules = star.flatMap((g) => g.rules);\n    }\n}\nlet best = null;\nfor (const r of rules) {\n    if (!ruleMatches(r.path, path)) continue;\n    const len = r.path.length;\n    if (!best || len > best.path.length || (len === best.path.length && r.type === 'allow')) best = r;\n}\nconst allowed = !best || best.type === 'allow';\n```\n\nNote the fallback: a bot with its own group ignores the `*` group entirely. Adding `User-agent: GPTBot` with a single `Disallow: /private/` means GPTBot no longer sees anything you wrote under `*`.\n\nOne more layer sits before robots.txt: your firewall or CDN. A WAF rule that blocks unknown bots can reject AI crawlers before they read a single line. If your logs show no hits from these tokens at all, check there too.\n\nMost AI crawlers fetch the raw HTML and parse it. They do not run your JavaScript. If your page is a client-rendered shell, they get an empty `<div id=\"root\">`.\n\nFetch the page with a crawler user agent and count the words that are actually in the HTML:\n\n```\nUA=\"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot\"\n\ncurl -s -A \"$UA\" https://yoursite.com/pricing \\\n  | perl -0pe 's/<script.*?<\\/script>//gs; s/<style.*?<\\/style>//gs; s/<[^>]+>/ /g' \\\n  | wc -w\n```\n\nIt is crude (navigation and footer count too), but the order of magnitude tells the story. A few dozen words on a page that looks full in the browser means the content arrives through JavaScript. The same command also shows whether your server or CDN answers this user agent with a 200 at all: add `-o /dev/null -w \"%{http_code}\\n\"`.\n\nThe checker uses the same idea with explicit thresholds: under 100 words, the page scores 0 on this check, and if the HTML also contains a framework root, it is flagged as rendered in the browser.\n\n``` js\nconst SPA_MARKERS = [\n    ['id=\"root\"', 'React'],\n    ['id=\"__next\"', 'Next.js'],\n    ['id=\"app\"', 'Vue or similar'],\n    ['id=\"__nuxt\"', 'Nuxt'],\n    ['ng-version', 'Angular'],\n    ['<app-root', 'Angular'],\n    ['id=\"svelte\"', 'Svelte'],\n    ['data-reactroot', 'React'],\n];\nconst likelyClientRendered = wordCount < 100 && (markers.length > 0 || /<script\\b[^>]*src=/i.test(html));\n```\n\nThe fix depends on the stack: server components in the Next.js App Router, SSR mode in Nuxt, pre-rendering or static generation for a React SPA. You can also do the manual test: View Page Source (not Inspect) and look for your headings.\n\n`llms.txt` is a Markdown file at the root of your site that summarises it and links the key pages. Jeremy Howard proposed the format on September 3, 2024, at llmstxt.org. The only required part is an H1 with the site name.\n\n```\ncurl -s -o /dev/null -w \"%{http_code} %{content_type}\\n\" https://yoursite.com/llms.txt\n# 200 text/plain; charset=utf-8   <- what you want\n```\n\nThe classic failure is a site that answers every unknown path with its home page: you get a 200, but with `text/html`. The checker treats that as \"no llms.txt\".\n\nNow the honest part. No major AI engine has confirmed using llms.txt to decide which brands to cite. Google says you don't need to create new machine readable files, AI text files, or markup to appear in AI Overviews and AI Mode. OpenAI points site owners to robots.txt and OAI-SearchBot for ChatGPT search. Where llms.txt is used is inference on documentation: OpenAI publishes its own docs index as an llms.txt file, and AI coding tools read these files to land on the right page.\n\n| File | What it does | Effect on AI visibility | \n|---|---|---|\n| `robots.txt` | Allows or blocks crawlers | Direct: a blocked crawler cannot cite you | \n| `sitemap.xml` | Lists the URLs you want indexed | Indirect: helps pages get indexed | \n| `llms.txt` | Summarises the site, links key pages | None confirmed by a major AI engine | \n\nSo: write one if you have docs, an API or many pages (it takes minutes), and check robots.txt first.\n\nI packaged these checks as a free Apify Actor, [AI Visibility Checker](https://apify.com/elraphh/ai-visibility-checker). Plain HTTP requests, no AI calls, no third party login. You paste up to 100 URLs per run and get a score out of 100 with one sentence per fix:\n\n| Check | Points | \n|---|---|\n| AI search and assistant crawlers in robots.txt | 25 | \n| Text readable without JavaScript | 20 | \n| Structured data (JSON-LD) | 15 | \n| XML sitemap | 7 | \n| AI training crawlers in robots.txt | 5 | \n| Indexable, title, meta description, canonical, llms.txt | 5 each | \n| `<html lang>` | 3 | \n\nTraining crawlers weigh little on purpose: blocking them is a valid choice. robots.txt, llms.txt and the sitemap are fetched once per domain, even if you check many pages of the same site.\n\n```\n{\n  \"urls\": [\"https://example.com\", \"https://example.com/pricing\"],\n  \"timeoutSecs\": 15\n}\n```\n\nA useful pattern is to schedule it after each deploy, so a robots.txt change or a rendering regression shows up the same day instead of months later.\n\nFor the per-bot details (user agent strings, published IP ranges, how to spot a fake GPTBot in your logs), see the [AI crawlers robots.txt guide](https://howseen.ai/blog/ai-crawlers-explained-gptbot-claudebot-perplexitybot-robots-txt?utm_source=devto&utm_medium=referral&utm_campaign=o2-syndication).\n\nSee what AI says about your brand: [howseen.ai](https://howseen.ai/?utm_source=devto&utm_medium=referral&utm_campaign=o2-syndication)", "url": "https://wpnews.pro/news/can-ai-crawlers-read-your-site-robots-txt-llms-txt-and-javascript-rendering", "canonical_source": "https://dev.to/raphaelaubry/can-ai-crawlers-read-your-site-robotstxt-llmstxt-and-javascript-rendering-checked-17mn", "published_at": "2026-11-06 04:41:37+00:00", "updated_at": "2026-11-06 04:54:36.723677+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "agent-protocols"], "entities": ["Raphaël Aubry", "Howseen", "OpenAI", "Anthropic", "Perplexity", "GPTBot", "OAI-SearchBot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/can-ai-crawlers-read-your-site-robots-txt-llms-txt-and-javascript-rendering", "markdown": "https://wpnews.pro/news/can-ai-crawlers-read-your-site-robots-txt-llms-txt-and-javascript-rendering.md", "text": "https://wpnews.pro/news/can-ai-crawlers-read-your-site-robots-txt-llms-txt-and-javascript-rendering.txt", "jsonld": "https://wpnews.pro/news/can-ai-crawlers-read-your-site-robots-txt-llms-txt-and-javascript-rendering.jsonld"}}