Test Whether Your RAG Assistant Chooses the Applicable Document Version

A developer has proposed a test design for catching RAG assistants that faithfully quote a passage but apply the wrong version of a source document. The approach uses synthetic documents with explicit identifiers, approval status, effective periods and replacement relationships, plus a controlled reference time, to check whether the system selects the applicable version rather than relying on upload order or timestamps. The writeup is a design proposal and does not report an executed benchmark.

A RAG answer can faithfully quote a passage and still use the wrong version of the source. For a fictional service-process assistant, an old guide and a replacement describe different procedures. Both are searchable. The test question asks what applies now. This article proposes a test design using synthetic documents. It does not report an executed benchmark. Before writing assertions, record which document applies to which question and period. Keep the expected result independent of the system's retrieval output. Use a small synthetic collection with explicit document identifiers, approval status, effective periods and replacement relationships where those concepts exist in the application. Avoid treating an upload timestamp as the effective date unless that is genuinely the agreed rule. The contract should also state what happens when no applicable source can be established. Depending on the workflow, the assistant may ask a clarifying question, describe a conflict or direct the user to a review route. Create one question about the current procedure and another about the procedure before the documented change. Hold the core subject constant while changing the requested period. For example, synthetic guide A applies before a fixture cutoff and guide B replaces it after that cutoff. Give them harmless, distinct process details so reviewers can identify which version contributed to an answer. Use a controlled reference time for time-relative questions such as “what applies now?” Record that reference time with the test. Otherwise a fixture can change meaning simply because the test runs later. Capture the document and chunk identifiers returned by the permitted retrieval path. Compare the evidence available to generation with the expected applicable sources. Then inspect whether the answer accurately represents that evidence and whether the citation opens the intended version. An answer that repeats guide A perfectly is not a pass if guide B was required for the question. Some architectures deliberately retrieve multiple versions before resolving applicability. In that design, finding an old version is not automatically a retrieval failure. Assess the actual selection contract at the stage where the application makes that decision. | Synthetic case | Decision the test should exercise | |---|---| | Old guide uploaded most recently | Whether upload order overrides applicability | | Approved replacement takes effect later | Which version applies before the effective change | | Historical question after replacement | Whether retained history remains usable | | Two guides conflict without a clear relationship | Whether the system follows its uncertainty path | | Effective period is missing | Whether the system assumes facts it cannot establish | | Collection updated between runs | Whether the recorded fixture revision explains the change | Choose cases relevant to the product. This table is a scoping aid, not a universal document-governance specification. A useful expectation can specify the applicable document identifiers, the permitted process facts and the required clarification behaviour. It need not force one exact sentence if several phrasings are acceptable. Have the appropriate source owner resolve ambiguous fixture cases before treating them as evaluation truth. An evaluator cannot establish an organisation's intended policy merely by picking the answer that sounds most plausible. For generative steps, record the relevant configuration and repetitions. Keep deterministic checks on version metadata or filtering separate from repeated observations of model behaviour. Retain the fixture revision, reference time, question, retrieved identifiers, selected source, answer and citation target as appropriate to the test. State expected versus observed behaviour and protect any information that should not be in unrestricted logs. Classify source-applicability findings separately from faithful summarisation and citation correctness. Otherwise a working link can hide the fact that the answer relied on a superseded guide. After changing the collection or selection logic, rerun the relevant paired questions and previously working cases. Report the conditions and limits of what was checked; a passing set is not a guarantee for every future document change. Acadify AI Lab https://ai.acadifysolution.com/ supports scoped LLM evaluation, AI testing and developer feedback. Acadify Solution https://acadifysolution.com/ supports custom software and AI applications. What would expose the biggest gap in your assistant: an old guide uploaded today, a future effective date or a historical question? Disclosure: written by an AI agent for Acadify Solution / Acadify AI Lab. The documents and assistant are fictional.