{"slug": "llms-txt-on-a-4-language-static-site-how-we-generate-it-validate-it-and-keep-it", "title": "llms.txt on a 4-language static site: how we generate it, validate it and keep it honest", "summary": "The team behind veilvex.com, a four-language Astro static site, generates its llms.txt and robots.txt files at build time from the same i18n content files that render the pages, so prices and descriptions in the machine-readable files cannot drift from the visible site. The build script fails the deploy when llms.txt exceeds a size limit, contains placeholder tokens, or shows a price that does not match the prices.ts source, and the team verified with real requests that GPTBot, ClaudeBot and PerplexityBot receive a 200 and full HTML despite Cloudflare's bot settings. Sitemap lastmod dates are taken from the last git commit touching each page's source, and a post-deploy script sends only new or changed URLs to IndexNow.", "body_md": "AI assistants now answer \"who does X in Y\" questions directly, and they build those answers from what they can read about you. For a small team, the cheapest thing you control is your own site: what crawlers are allowed to read, and how clearly your facts are stated.\n\nOn veilvex.com (an Astro static site in English, Russian, Azerbaijani and German) we treat this as part of the build, not a one-off file. Here is the setup and the rules we ended up with.\n\nMany sites block AI crawlers by accident: a template, a CDN toggle, a security plugin. We generate `robots.txt` at build time and list the crawlers by name, even though `User-agent: *` already allows everything:\n\n```\nUser-agent: *\nAllow: /\nContent-Signal: search=yes, ai-input=yes, ai-train=yes\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /\n```\n\nWhy list them? Because the next person who adds a \"block bots\" rule will see the intent, and because a test can assert that every agent in the list stays allowed. The `Content-Signal` line follows [contentsignals.org](https://contentsignals.org) and states how the content may be used.\n\nAlso check your CDN: on Cloudflare, \"Block AI bots\" and bot-fight settings can override a perfectly good `robots.txt`. We verified with real requests that GPTBot, ClaudeBot and PerplexityBot get a 200 and the full HTML.\n\n[llms.txt](https://llmstxt.org) is a proposed convention: a short Markdown file at the site root that tells language models what the site is and where the important pages are. It is not a ranking factor and no engine promises to use it. We still ship it because it costs nothing and removes ambiguity.\n\nThe important decision: **never write it by hand.** Ours is built by a script from the same i18n content files that render the pages. So the price on the services page and the price in `llms.txt` cannot drift apart.\n\nWhat goes in (English, compact):\n\n`Organization.description` in JSON-LD and external profiles);\nNext to it there is `llms-full.txt` with the full text of every page in all four languages (about 400 KB for us), for tools that want everything in one fetch.\n\nThis is the part that keeps it honest. The script throws, and the deploy stops, when:\n\n`llms.txt` grows over `prices.ts`);`[[...]]` or `{{...}}`.\nA simplified version of the price check:\n\n``` js\nfor (const s of services) {\n  const expected = formatFrom(PRICES[s.id]);\n  if (!s.line.includes(expected)) {\n    throw new Error(`llms.txt: ${s.id} shows a price that is not ${expected}`);\n  }\n}\n```\n\nModels repeat what they read. If your site says \"guaranteed first place\" in one corner, that is what an assistant may quote back to a customer. A build-time check is cheaper than finding it later.\n\nEach service page carries `Service` + `Offer` and `FAQPage` JSON-LD built from the same content object as the visible text, with stable `@id` s across languages. The organization node uses the same brand line as `llms.txt`. One source, three outputs (HTML, JSON-LD, llms.txt), so a crawler sees the same facts whichever it reads.\n\n`<lastmod>` is the date of the last git commit that touched the page's source files, not the build time. A build date on every URL tells crawlers nothing. After a deploy, a script compares the new sitemap with the live one and sends only new or changed URLs to IndexNow (Bing, Yandex and others).\n\nNone of this makes an assistant recommend you. It removes the reasons not to: blocked crawlers, conflicting prices, vague descriptions. Being named still depends on being mentioned in places models trust: directories, reviews, articles, discussions. We check monthly by asking ChatGPT, Perplexity and Gemini the same set of queries and logging who they name.\n\n`robots.txt` and not blocked by your CDN`lastmod` dates, IndexNow after deploy\nYou can see the live files at [veilvex.com/llms.txt](https://veilvex.com/llms.txt) and [veilvex.com/robots.txt](https://veilvex.com/robots.txt). Questions about the setup are welcome in the comments.\n\n*We are [Veilvex](https://veilvex.com), a remote team working on GEO (visibility in AI answers), SEO, websites and AI automation.*", "url": "https://wpnews.pro/news/llms-txt-on-a-4-language-static-site-how-we-generate-it-validate-it-and-keep-it", "canonical_source": "https://dev.to/veilvex/llmstxt-on-a-4-language-static-site-how-we-generate-it-validate-it-and-keep-it-honest-4m8f", "published_at": "2026-10-29 11:11:13+00:00", "updated_at": "2026-10-29 11:16:57.964623+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "structured-data", "generative-engine-optimization", "ai-search"], "entities": ["veilvex.com", "Cloudflare", "GPTBot", "ClaudeBot", "PerplexityBot", "Google-Extended", "IndexNow", "Astro"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llms-txt-on-a-4-language-static-site-how-we-generate-it-validate-it-and-keep-it", "markdown": "https://wpnews.pro/news/llms-txt-on-a-4-language-static-site-how-we-generate-it-validate-it-and-keep-it.md", "text": "https://wpnews.pro/news/llms-txt-on-a-4-language-static-site-how-we-generate-it-validate-it-and-keep-it.txt", "jsonld": "https://wpnews.pro/news/llms-txt-on-a-4-language-static-site-how-we-generate-it-validate-it-and-keep-it.jsonld"}}