cd /news/ai-tools/ai-translation-pipeline-validators-a… · home › topics › ai-tools › article
[ARTICLE · art-171738] src=dev.to ↗ pub= topic=ai-tools verified=true sentiment=↑ positive

AI Translation Pipeline: Validators Around a gpt-4o Call

A developer built an AI translation pipeline for the pi-pi.ee e-commerce shop that generates 27 locales from a Russian source file using gpt-4o, wrapping the model call in validators rather than relying on prompt engineering. The pipeline diffs missing keys so existing translations are never retranslated, and runs ICU, argument, tag and plural checks at both write time and commit time via a shared i18n-check.ts implementation, exiting with code 1 and a list of omitted strings when validation fails.

recommendation

Validate every model-returned translation with a shared ICU and plural checker before writing it, and diff only missing keys so hand-edited strings survive reruns.

more in playbook: Verifying model output
by read10 min views6 publishedNov 18, 2026

An AI translation pipeline needs checks that can say ‘no’ more than it needs a better prompt. In the script that fills the 27 target languages of pi-pi.ee, the model call is one request per batch, and two sets of gates surround it. At write time: a diff of missing keys, a check on whether the reply was cut off, an ICU and plural validator that must pass before a string is written, and exit code 1 with a list of what was left out. At commit time: i18n:check and glossary:check, which run as part of npm run lint.

Below is that code as it stands, the real defects behind some of the gates, and a section on what the gates still do not catch. Routing, hreflang and formatting are in Next.js i18n with 30 Languages; I will not repeat them.

The shop has a Russian source file and 27 generated locales. The first version of any such script sends strings to a model and saves the reply. It works on the first run and fails later, quietly:

A prompt is a request, and these are hard to ask a model to detect in its own output. Code that runs after the call and is allowed to refuse is a better place for them.

translate.py never translates a whole file. It computes the keys that exist in ru.json and not in the target, and sends only those:

for key in source:
    if key not in target:
        missing.append((current_path, source[key]))
    elif isinstance(source[key], dict) and isinstance(target[key], dict):
        missing.extend(get_missing_keys(source[key], target[key], current_path))

An existing translated string is never retranslated. That is the property that lets a person edit a locale file by hand and have it survive the next run. Keys that exist in a locale file but not in the source are removed on every run, except the dotted paths listed in scripts/locale-overrides.json; hand-written, locale-specific content has to be registered there or the next run deletes it.

The same diff powers --dry-run: it prints missing, obsolete and invalid keys per locale, with no API call, no writes and no API key. It also validates the strings already in the files, so it checks the current state as well as previewing the next run.

The cost of diffing is a rule people have to follow. translate.py only fills missing keys, so an edit to an existing Russian string leaves the other locales stale until the documented procedure is followed: delete that key from every other locale and rerun. That is a procedure, not an enforced check, and it comes back in the last section.

Every string the model returns goes through a validator before it is placed in a file. The validator is scripts/i18n-check.ts, the same code that npm run i18n:check runs over the whole catalogue at commit time, called by the Python script in --validate mode:

invalid = {f["key"]: f["error"] for f in validate_translations(candidates)}
for candidate in candidates:
    key = candidate["key"]
    if key in invalid:
        failures.append(f"{key}: {invalid[key]}")
    else:
        all_translated[key] = candidate["target"]

Using one implementation for both means the write-time check and the commit-time check cannot drift apart: there is one ICU parser and one source of plural data.

For each source and target pair the validator runs these checks:

// Simplified from validate() in scripts/i18n-check.ts
const want = shapeOf(parse(source));
const got = shapeOf(parse(target)); // a parse error is returned as a failure
if (sorted(want.args) !== sorted(got.args)) errors.push(`arguments differ from source`);
if (sorted(want.tags) !== sorted(got.tags)) errors.push(`tags differ from source`);
return [...errors, ...pluralErrors(locale, got)];

The target must parse as ICU. It must use the same {arguments} and the same <tags> as the source. And every plural must contain the required CLDR categories of the target language, which come from Intl.PluralRules: other always, and for example one, few and many for Russian. In French, Italian, Spanish and Portuguese many is treated as optional. A model copying the source structure can get this wrong. One exception: strings containing raw HTML tags, such as FAQ answers, skip this ICU check; more on that below.

A failing string is not written. The other strings in the batch that passed are written, the file is saved, and at the end the run prints the list of keys that were left out and exits with code 1. Because those keys are still missing, the next run picks them up through the same diff. Nothing marks a failure as ‘done’.

The validator also feeds the prompt. Before a request, the script asks it for the target locale’s plural categories with sample numbers and writes them into the system prompt. The model is told the rule that will be enforced afterwards. That should help the first pass; the validator is what stops a miss from being written.

Batches are cut at 50 strings or about 10,000 characters. The character limit exists because the reply is capped at the model’s output limit: if it is reached mid-object you do not get 40 good strings and 10 missing ones, you get a broken JSON object and lose the whole batch. A long HTML answer should not share a call with 49 other strings.

The script checks finish_reason on every response:

if response.choices[0].finish_reason == "length":
    failures.extend(f"{key}: API response truncated at the output limit" for key in batch)
    continue

A truncated reply fails the batch with its key list; it is not parsed ‘as far as it goes’. An empty reply, an unparseable one and a key missing from a valid reply are each reported separately. The call is gpt-4o at temperature 0.3. That makes output steadier, not reproducible, which is another reason the gates sit after the call. The cost side of production model calls is a separate topic.

Batching creates the glossary problem. The shop sells two products: waterless urinals and waterless outdoor sinks. Each batch re-decides what to call them in the target language. Over a few runs, one locale ends up with several words, and some of them mean something else. The defects I documented in review:

Locale The model produced What it actually means
et uriin urine
lv urīns urine
hr vodokotlić toilet cistern
pl zlew kitchen sink
uk писуар misspelling of пісуар

A person found these by reading the output in review; the pipeline did not catch them. Afterwards I turned each one into a check.

scripts/glossary.json holds one entry per locale: the approved term for each product, the qualifier that term needs, forbidden terms with a plain statement of what each one really is, the script, the way to address the reader, and the title separator. It records established usage, counted in the files; it does not invent a terminology policy.

The glossary is used in two places. At write time, translate.py renders the locale’s entry into the system prompt, last, so it overrides looser hints above it. It does not check the glossary on the returned strings. That happens at commit time: glossary:check scans every message file, as part of npm run lint. Two kinds of check run there.

The negative check is the obvious one: a forbidden stem, or text in the wrong alphabet, fails. A stem also matches up to four inflection letters, so zlew catches zlewy but not an unrelated word that merely starts the same way.

The negative check has a hole, and I fell into it. A forbidden list only knows the words someone already listed. On 20 September 2026 it was green while six locales carried the kitchen-sink word on the sink product’s H1: fregadero (es), sudoper (hr), drez (sk), mosogató (hu), gootsteen (nl) and lavello (it). Nobody had listed them. I found them by dumping that one key across all 28 locales and reading the column.

So the check now also runs in the opposite direction, on what I call identity surfaces: strings that should name the product, matched by key path, so no language parsing is needed.

// Simplified from scripts/glossary.ts
const IDENTITY_SURFACES = [
  /^(sinks|urinals)\.name$/,
  /^(sinks|urinals)\.seo\.title$/,
  /^(sinks|urinals)\.hero\.header$/,
  /^(sinks|urinals)\.products\.[^.]+\.header$/,
  // ...
];

if (slot && !mentionsTerm(text, entry.terms[slot], false)) {
  console.error(`Tier-1 surface does not use an approved ${slot} term`);
  failures++;
}

On those paths the check fails unless the string contains an approved term for the slot (matched by stem, or by compound head). Two more rules are narrower than they sound. A sink noun needs its qualifier (‘garden’, ‘outdoor’ and so on) only where the page does not already set the category: the check passes automatically for anything under sinks.*, for .name labels, and for a string that also contains the urinal noun. In practice that leaves strings outside the sink subtree, such as the homepage catalogue header. And for seo.title the check flags a dash or pipe that differs from the locale’s recorded separator; it does not require a separator to be present, and the SEO prompt itself asks for none. The key path says which product slot a string belongs to, which keeps false positives low; a rule that fires on legitimate copy gets ignored.

Strings that drive search results are not translated like button labels. Keys matching patterns such as .seo., .header, the homepage hero and FAQ entries get a different prompt, with the locale’s keyword spec: a file per locale with Tier 1 terms from Keyword Planner volumes, negative lists and disambiguation rules. The raises if a spec has no ‘Tier 1’ section:

if "Tier 1" not in spec:
    raise ValueError(f"{path.name}: no Tier 1 section found — the spec format changed")

Same idea, small scale: if someone reformats the spec and the parser silently finds nothing, every SEO key is translated without keywords and no error. A locale with no spec file is allowed, but the run prints a warning.

All of the above checks shape and terminology. None of it can tell whether a sentence sounds like a person wrote it.

That is a separate pass that I run; translate.py does not call it. It uses two Claude Code agent roles defined in the repository, localization-reviewer and proofreader. The reviewer compares the translation with the source and checks buyer terminology; the proofreader does the final language pass. A loop limit stops them bouncing a text back and forth: it goes back once, and then the reviewer’s linguistic verdict is final or the disagreement comes to me. The project procedure also asks for a manual spot-check of a few major languages after a run. The output of the agents is a model’s judgement, not automated verification.

The validators do not prove fluency or tone. They remove mistakes that should never need a reader’s attention, so the attention goes to what only a reader can find. A reviewing agent is a second opinion, not a guarantee.

i18n-check.ts skips the ICU, placeholder and tag checks for strings that contain HTML tags, such as FAQ answers, because they are rendered as HTML and not as ICU. Those get key parity only.IDENTITY_SURFACES. Elsewhere the only automated guard is the forbidden list, which knows only what someone has added.translate.py only fills missing keys. An edited Russian string leaves the other locales stale until the delete-and-rerun procedure is followed, and I found no check in these scripts that detects it.ru.json says. That limits what I claim. The write-time gates stop strings that are broken in shape from being written; the commit-time checks flag known-wrong and missing product terms. Neither makes a translation good.

A translation management system offers review workflows, translation memory and a place for translators, which this script lacks. I did not evaluate one for this shop, so this is not an argument against them. The reason to write the gates yourself is narrower: they are written in the shop’s own terms (key paths, product slots, the words that mean the wrong thing) and the commit-time ones run with the rest of the project’s lint. For a few hundred strings in two languages, a script like this is more than you need.

finish_reason, and size batches so one truncation cannot take everything. slug="ai-integration"

text="Adding a model call to a content or data flow? I design what it is allowed to write: validators, refusals and a review step that is honest about its scope."

/>

Further reading:

── more in #ai-tools 4 stories · sorted by recency
── more on @pi-pi.ee 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/ai-translation-pipel…] indexed:0 read:10min 2026-11-18 · —