# llms.txt on a 4-language static site: how we generate it, validate it and keep it honest

> Source: <https://dev.to/veilvex/llmstxt-on-a-4-language-static-site-how-we-generate-it-validate-it-and-keep-it-honest-4m8f>
> Published: 2026-10-29 11:11:13+00:00

AI assistants now answer "who does X in Y" questions directly, and they build those answers from what they can read about you. For a small team, the cheapest thing you control is your own site: what crawlers are allowed to read, and how clearly your facts are stated.

On veilvex.com (an Astro static site in English, Russian, Azerbaijani and German) we treat this as part of the build, not a one-off file. Here is the setup and the rules we ended up with.

Many sites block AI crawlers by accident: a template, a CDN toggle, a security plugin. We generate `robots.txt` at build time and list the crawlers by name, even though `User-agent: *` already allows everything:

```
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /
```

Why list them? Because the next person who adds a "block bots" rule will see the intent, and because a test can assert that every agent in the list stays allowed. The `Content-Signal` line follows [contentsignals.org](https://contentsignals.org) and states how the content may be used.

Also check your CDN: on Cloudflare, "Block AI bots" and bot-fight settings can override a perfectly good `robots.txt`. We verified with real requests that GPTBot, ClaudeBot and PerplexityBot get a 200 and the full HTML.

[llms.txt](https://llmstxt.org) is a proposed convention: a short Markdown file at the site root that tells language models what the site is and where the important pages are. It is not a ranking factor and no engine promises to use it. We still ship it because it costs nothing and removes ambiguity.

The important decision: **never write it by hand.** Ours is built by a script from the same i18n content files that render the pages. So the price on the services page and the price in `llms.txt` cannot drift apart.

What goes in (English, compact):

`Organization.description` in JSON-LD and external profiles);
Next to it there is `llms-full.txt` with the full text of every page in all four languages (about 400 KB for us), for tools that want everything in one fetch.

This is the part that keeps it honest. The script throws, and the deploy stops, when:

`llms.txt` grows over `prices.ts`);`[[...]]` or `{{...}}`.
A simplified version of the price check:

``` js
for (const s of services) {
  const expected = formatFrom(PRICES[s.id]);
  if (!s.line.includes(expected)) {
    throw new Error(`llms.txt: ${s.id} shows a price that is not ${expected}`);
  }
}
```

Models repeat what they read. If your site says "guaranteed first place" in one corner, that is what an assistant may quote back to a customer. A build-time check is cheaper than finding it later.

Each service page carries `Service` + `Offer` and `FAQPage` JSON-LD built from the same content object as the visible text, with stable `@id` s across languages. The organization node uses the same brand line as `llms.txt`. One source, three outputs (HTML, JSON-LD, llms.txt), so a crawler sees the same facts whichever it reads.

`<lastmod>` is the date of the last git commit that touched the page's source files, not the build time. A build date on every URL tells crawlers nothing. After a deploy, a script compares the new sitemap with the live one and sends only new or changed URLs to IndexNow (Bing, Yandex and others).

None of this makes an assistant recommend you. It removes the reasons not to: blocked crawlers, conflicting prices, vague descriptions. Being named still depends on being mentioned in places models trust: directories, reviews, articles, discussions. We check monthly by asking ChatGPT, Perplexity and Gemini the same set of queries and logging who they name.

`robots.txt` and not blocked by your CDN`lastmod` dates, IndexNow after deploy
You can see the live files at [veilvex.com/llms.txt](https://veilvex.com/llms.txt) and [veilvex.com/robots.txt](https://veilvex.com/robots.txt). Questions about the setup are welcome in the comments.

*We are [Veilvex](https://veilvex.com), a remote team working on GEO (visibility in AI answers), SEO, websites and AI automation.*