Test multilingual AI search by pairing real business questions with the exact source passages that should support them, then vary the query and document languages deliberately. Score retrieval before asking a model to write the answer. A fluent response in the requested language can hide that the system found the wrong policy, missed a local exception or relied on a document the user could not access.
- 01Separate language directionsA query in one language retrieving another is its own test condition.
- 02Judge passages, not fluencyInspect the evidence before evaluating the generated answer.
- 03Keep entities stableNames, codes and local terms need explicit acceptance checks.
- 04Report limits honestlyA language list and a small test set do not establish universal accuracy.
01 — Search directionDefine the actual cross-language task #
A business knowledge base may contain English manuals, local-language policies and customer questions in several languages. Those combinations create different tasks. An English question retrieving a Spanish document is not the same condition as a Spanish question retrieving an English document, and neither is identical to searching within Spanish.
Write down the language direction and the intended business outcome for each test. A hypothetical support assistant might need to find an English product specification from a German question while preferring a German regional warranty policy when that policy governs the customer. The system must distinguish language availability from applicability; translating the query does not decide which policy is authoritative.
The RAG business guide explains the overall retrieval-and-answer pattern. Multilingual testing should preserve that separation. First establish whether the correct evidence was retrieved, then assess whether the answer expresses it accurately in the requested language.
Keep locale and language as separate fields in the test design. An English question from one region may need a different policy from an English question elsewhere, while a French question may legitimately seek the same global specification. A language detector cannot establish the governing region or product version. Record those business constraints explicitly so a successful translation does not route the request to an inapplicable but linguistically convenient document.
Query and source align
Establish whether the document can be found before adding a language boundary.
Query crosses to source
Test each required language direction with judged supporting passages.
Names and terms span languages
Preserve entities and local terminology while interpreting the question.
02 — Answer keyBuild matched questions and evidence passages #
Choose questions with identifiable supporting passages and record why each passage is relevant. Include an exact specification, a condition or exception and a question that requires more than one source. Also include questions the corpus cannot answer. Otherwise, the evaluation rewards systems that always return something plausible.
Use fluent reviewers to create or approve equivalent query versions. Literal translation can distort the way a customer would ask the question, while free rewriting can change the task. Preserve the intended information need and allow natural phrasing. Store the original query, the approved variant and any nuance that affects relevance.
For a hypothetical warranty question, the answer key might require both the duration and the regional exclusion. A document that mentions the product but omits the exclusion is partially relevant, not sufficient evidence. This distinction matters because a multilingual retrieval system can appear successful when it finds the right topic but misses the condition that changes the answer.
Avoid translating every source into the query language merely to simplify the answer key. That would remove the cross-language retrieval condition the test is supposed to examine. Review the original source and record the relevant passage with a reviewer-approved explanation of its meaning. If a translated source is itself part of the production corpus, treat it as a distinct version with provenance and check whether it remains aligned with the governing original.
These are proposed evaluation cases. The guide does not report a completed multilingual benchmark or accuracy measurements for any model.
03 — Representation contractKeep the embedding setup consistent #
Record the exact embedding model, dimensions, preprocessing and query/document configuration used by the index. A model name without those details is not enough to reproduce a retrieval result. Keep the same document snapshot while comparing language conditions so a source update does not masquerade as a language improvement.
Voyage's retrieval guidance, checked October 11, 2026, recommends distinguishing query and document inputs for retrieval. That is a concrete setup detail to preserve when evaluating the relevant models. The same documentation includes compatibility statements for particular model families; do not generalize them to unrelated models because their vectors happen to have matching dimensions.
A provider's supported-language list helps decide what to test, but it is not an acceptance result for your documents. Domain vocabulary, mixed scripts, document quality and query style can all affect the assembled search path. Keep candidate comparisons bounded to the exact configuration and workload you observed.
Check truncation and extraction before interpreting language differences. A parser may preserve one script well and corrupt another, or a long document may lose the relevant passage before embedding. Inspect the actual text entering the model for each document class. When that text is incomplete, the experiment is measuring preprocessing quality as much as multilingual representation, and the repair should address the missing information before changing models.
- Version the model, preprocessing and input-type settings together.
- Keep the corpus snapshot stable during a comparison.
- Follow explicit compatibility documentation rather than vector length alone.
04 — Retrieval evidenceInspect candidates before asking for an answer #
Run the queries through retrieval and save the candidate identifiers, passages and ranks. Review whether the required evidence is present within the context budget the application actually uses. A relevant document at a rank the answer model never receives is not a successful end-to-end retrieval result.
Separate missing evidence from poor ordering. If the governing passage is absent, inspect ingestion, language handling, filters and first-stage search. If it is present but buried beneath broad overviews, a reranker may help. Our reranker diagnosis guide explains that distinction without assuming every failure requires a new embedding model.
A reranker still needs evaluation across the required language pairs. Its score does not establish that a passage preserves the intended meaning or satisfies the business question. Keep reviewer judgments and model scores separate, and do not invent a universal cutoff that supposedly works for every language and document type.
Choose the retrieval budget according to the application, then report it with the result. Finding a passage somewhere in a very large candidate set is weaker than finding it inside the context budget the answer model receives. Keep the same budget when comparing query variants unless the experiment explicitly tests a different budget. Otherwise, a language path can appear better simply because it was allowed to return much more material.
| Proposed test conditions and inspection targets, not results from a completed model comparison. | ||
|---|---|---|
| Test condition | Evidence to inspect | Typical hidden failure |
| --- | --- | --- |
| Same-language query | Required passage in candidates | Baseline ingestion problem |
| Cross-language query | Equivalent evidence at usable rank | Topic match without the exception |
| Mixed-language query | Names and codes preserved | Entity rewritten or ignored |
| Unanswerable query | No unsupported answer produced | Plausible but irrelevant evidence |
05 — Entity fidelityChallenge names, codes and local terminology #
Product codes, proper names and local terms often carry more business meaning than the surrounding sentence. Include queries where these elements remain unchanged across language variants, and queries where the accepted local equivalent differs. A system that translates a product identifier into ordinary words may retrieve a semantically related but incorrect item.
In a hypothetical parts catalogue, a short code distinguishes two similar components. Test the code alone, inside a natural-language question and inside a mixed-language question. Inspect whether lexical and semantic paths preserve the distinction. The hybrid search reference explains why exact-term retrieval can complement vector search in such cases.
Include accents and script variants only where they reflect the real workload. Do not assume every spelling difference is harmless or every transliteration refers to the same entity. The answer key should identify acceptable aliases and unresolved ambiguity. When the evidence permits multiple products, the correct response may be a clarification rather than a confident selection.
Include terms that look similar across languages but differ in meaning where they occur in the business domain. A fluent query can contain a word that points to a different concept in another corpus language. A reviewer should identify the intended sense and acceptable evidence before testing. The system may need clarification when the context is insufficient; confidently choosing the familiar-looking term should not receive credit merely because the returned passage shares vocabulary.
Save the original entity, the query variant, the retrieved entity and the business consequence. “Poor multilingual search” is too broad to guide a repair.
06 — Alternative pathCompare query translation as a separate experiment #
Translating a query into a corpus language can be a useful candidate approach, but it adds another transformation. Keep the original query and the translated version, then inspect whether names, conditions and negation survive. Do not assume that fluent translation preserves the exact retrieval intent.
Compare direct multilingual retrieval with the translated-query path using the same sources and relevance judgments. If the translated path finds better evidence for some questions, identify which ones and why. It may help with domain terminology while harming a product code or local policy phrase. A single aggregate result can hide that tradeoff.
The translation QA guide separates fluent wording from preserved business meaning. Apply the same principle before a translated query reaches search. Translation is a testable component in the pipeline, not a universal repair for weak cross-language embeddings or missing source material.
Keep translated-query caching scoped to the original meaning and relevant settings. A cached transformation that ignores locale, product version or an updated glossary can repeatedly send the wrong search request even after the retrieval model improves. Record the transformation version and inspect cache hits during evaluation. This makes it possible to distinguish a model failure from reuse of an obsolete query interpretation that never reached the current translation component.
- Preserve the original and transformed query for review.
- Check identifiers, negation and conditions before comparing results.
- Report where translation helps and where it introduces new errors.
07 — Answer verificationScore answers only after the evidence passes #
Once retrieval provides the required passages, run the answer workflow and review the output in the requested language. Check whether it preserves the source's conditions and cites the evidence actually used. A correct retrieval result can still produce a misleading answer if the model smooths away a qualification or confuses similar documents.
Use fluent reviewers for meaning and naturalness, but give them the source passage and the business acceptance criteria. A reviewer who sees only the translated answer cannot know that a regional exception was omitted. Back-translation may help flag suspicious cases, but it does not independently prove equivalence. Keep authorization checks active throughout the evaluation. A cross-language query must not retrieve restricted evidence simply because its language differs from the source. Our document-permissions guide covers connector and cache boundaries. Test allowed and denied users with the same language variants so an apparent relevance gain does not conceal an access-control failure.
Review the source-language citation alongside the answer-language wording. A user may be able to understand the answer but not independently read the cited source, which makes faithful qualification especially important. Where the product offers a translated excerpt, label its relationship to the original and preserve a way to inspect the source. Do not let an assistant present its own translation as though it were an official localized policy published by the organization.
Score retrieval support, answer meaning and access control separately. One fluent answer should not erase a failure in any of those categories.
08 — Release scopePublish a narrow support claim you can defend #
Report the language directions, document classes, task types and conditions actually tested. Keep the number and composition of cases visible in the internal evaluation record without pretending a small sample represents every speaker or domain. A successful set of policy questions says little about scanned diagrams or specialized technical manuals unless those were included.
Keep difficult and unresolved cases as regression tests. When the embedding model, reranker, translation step or corpus parser changes, rerun the cases that establish the supported behavior. New source documents and terminology can also change the task, so the evaluation should evolve with the knowledge base rather than remain a launch-day artifact.
Our AI transformation service helps teams define useful support boundaries and observable tests for knowledge workflows. The practical goal is an assistant that finds the right authorized evidence across the language combinations your business needs, preserves its meaning and admits when the available sources do not settle the question.
Use separate outcome categories for missing evidence, wrong ordering, changed meaning and unauthorized material. Those categories point to different owners and repairs. An overall quality number can be useful for tracking a stable test set, but it should not replace the failure record. The release decision should explain which combinations are supported and which require clarification or human assistance, so operators can act on the evaluation rather than admire a score.
- State the tested language directions and document types.
- Preserve failure cases for later model and corpus changes.
- Keep an explicit path for ambiguity and unsupported questions.
Prove the evidence crosses the language boundary
Build matched query and passage pairs, inspect retrieval before generation and review the answer with the source in view. Keep names, conditions and permissions fixed throughout.
A multilingual search launch is ready when the required business questions work under those checks, not when a provider lists the relevant languages.