Why retrieval quality, document structure, and relevance matter more than adding a model to your search system
The model produced a confident answer, but the relevant document was never retrieved.
Consider a simple question: “Can a customer request a refund after the contract has been renewed?”
Imagine a knowledge base containing a general refund policy, a contract renewal procedure, and an older policy that has since been replaced. The system retrieves related information, passes it to the model, and generates an answer. But it misses the evidence that determines whether the answer is correct.
This is how a RAG system can fail without appearing broken. Search returns results. The model generates fluent text. The application responds successfully. Yet the answer is unsupported.
Retrieval-augmented generation (RAG) is easy to explain: index documents, retrieve relevant passages, and give them to a language model. Building a reliable system is harder because finding text about a topic is not the same as finding the evidence needed to answer a question.
The difficult engineering work often happens before generation.
A vector search might find a general refund policy while missing the renewal procedure that contains the decisive condition. The retrieved text may be semantically relevant without being sufficient to answer the question.
The model cannot reliably recover a rule it never sees. It may guess, combine unrelated conditions, or present an incomplete answer with unwarranted confidence.
This distinction changes how we debug the system. If retrieval is the failure point, changing the prompt or upgrading the model may increase cost without fixing the problem.
A RAG pipeline is more than retrieval and generation.
Document Sources → Ingestion → Chunking → Indexing → Retrieval → Filtering → Reranking → LLM Generation → Answer
Every stage can introduce a different failure:
Document sources: Missing, duplicated, or outdated documents.
Ingestion: Content extracted incorrectly.
Chunking: Rules and exceptions separated from one another.
Indexing: Missing or stale representations.
Retrieval: Relevant evidence not found.
Filtering: Useful or authorized evidence removed.
Reranking: Relevant passages ranked too low.
Generation: Evidence misinterpreted or unsupported claims added.
Figure 1. A RAG pipeline. Answer quality depends on every stage, not only the language model.
Suppose a refund policy states the general rule in one section and the renewal exception in another. A fixed-size split might separate the exception from the rule it modifies. Both chunks remain valid text, but neither may provide enough context when retrieved alone.
Very small chunks can lose references and conditions. Large chunks preserve context but introduce unrelated content, increasing the amount of text the model must process.
Chunk boundaries should reflect the source material:
Policies need to preserve conditions, exceptions, and section headings.
Contracts may need clause boundaries and cross-references.
Product documentation benefits from keeping version details near feature descriptions.
Tables need their headings, units, and values to remain associated.
Metadata such as document title, section, version, effective date, and source identifier can help preserve context without repeating entire documents in every chunk.
Consider a table containing product limits. If the indexing pipeline extracts a value but loses the column heading, the retrieved text may contain the correct number without explaining what it means. The model then has to infer a relationship that the original document already defined.
Chunking is therefore part of knowledge modeling, not just text preprocessing. The goal is to create units that are focused enough to retrieve and complete enough to interpret.
“Explain our refund policy” is different from “Does policy RF-204 apply after renewal?” The first may benefit from semantic matching; the second may depend on an exact identifier and a specific document version.
Hybrid retrieval combines keyword-based and vector search, then merges their results. Keyword search can capture critical terms such as “renewal” and exact identifiers, while vector search can tolerate differences in phrasing.
The value of hybrid search lies in combining these complementary strengths rather than assuming one retrieval method works equally well for every query.
Microsoft’s Hybrid Search Overview explains how text and vector search can work together.
The right retrieval strategy depends on the data and the questions users ask. A technical support system may need exact error-code matching. A policy assistant may depend more on semantic similarity, while still requiring exact policy identifiers and version constraints.
There is no reason to assume that one retrieval configuration is optimal for every knowledge domain.
A passage can be highly similar to a question while omitting the condition that determines the outcome. Another passage with a lower score may contain the decisive exception.
This creates several engineering questions:
How many candidates should retrieval return?
Should keyword and vector results be combined?
Which documents are authoritative?
How should document versions and effective dates affect ranking?
When should a reranker reorder the candidates?
What happens when the evidence is incomplete or contradictory?
A typical approach is to retrieve a broader set of candidates and then rerank them using a more detailed relevance assessment. This can improve the order of results, but it cannot recover evidence that the initial retrieval stage never returned.
Filtering also needs care. Removing obsolete or unauthorized documents is essential, but an overly broad filter can eliminate the only passage that answers the question.
For systems handling private or tenant-specific information, authorization must be enforced as part of retrieval. It should not depend on the language model choosing to ignore an unauthorized passage. Microsoft’s RAG information retrieval guidance provides additional guidance on retrieval design.
Consider: “Can this customer receive a refund under the current policy?”
A reliable answer may require three pieces of evidence:
The current refund policy.
The customer’s contract renewal date.
The applicable exception for that product or contract type.
Retrieving only the general policy is not enough. Retrieving the contract without the exception is not enough either.
This is where multi-step retrieval becomes relevant. The system may need to identify which facts are missing, retrieve additional documents, and combine the evidence before generating an answer.
However, additional retrieval steps introduce their own risks. An early mistake can lead subsequent searches in the wrong direction. The system may also accumulate irrelevant passages or treat two documents as consistent when they are not.
A useful design principle is to make evidence requirements explicit. For a question that depends on several conditions, define which facts must be established before an answer can be considered complete.
The system can then distinguish between a question that has sufficient evidence and one that still needs more information.
Sending an entire knowledge base into the prompt may increase latency, cost, and the amount of irrelevant information the model must process. Large inputs can also make it harder for a model to consistently use the most important details.
More context is not necessarily better context.
A good retrieval system narrows the evidence to the material that matters for the current question. It preserves enough surrounding information to interpret that evidence correctly, while avoiding unrelated passages.
The objective is not to minimize context at all costs. It is to provide the smallest sufficient set of trustworthy evidence.
That balance depends on the task. A short troubleshooting question may need only one document section. A complex contract question may need multiple clauses and their cross-references.
The context window is a resource. Retrieval is how the system decides what deserves that resource.
One useful starting point is Recall@K: whether the expected relevant source appears among the top K retrieved results.
If a question has one expected source and that source appears in the top five results, its source recall at five is 1. If it does not appear, the score is 0. Across a set of questions, the average indicates how often retrieval found the expected evidence.
For questions that require multiple documents, the evaluation should also check whether all required sources were retrieved.
Consider this small evaluation set:
QuestionExpected sourceFound in top 5?RankWhich refund policy applies after renewal?Current refund policyTo be evaluated — Which policy applies to product XJ-420?Product policyTo be evaluated — What happens when a subscription is cancelled mid-cycle?Subscription policyTo be evaluated —
These are illustrative evaluation questions, not measured results. Each question should have a known set of relevant source documents. Track whether those sources appear in the top-K results and how highly they rank.
A simple Python function can calculate source recall at K:
def retrieval_metrics_at_k(expected_sources, retrieved_sources, k=5):
"""
expected_sources: iterable of relevant source IDs
retrieved_sources: source IDs ordered by retrieval rank
"""
expected = set(expected_sources)
retrieved = retrieved_sources[:k]
if not expected:
return {
"recall_at_k": None,
"all_sources_found": None,
}
found = expected.intersection(retrieved)
return {
"recall_at_k": len(found) / len(expected),
"all_sources_found": found == expected,
}
For a single-source question, Recall@K is either 0 or 1. For a question requiring several sources, it measures the fraction of expected sources retrieved. The all_sources_found field indicates whether every expected source appeared in the top K.
This is only a starting point. Source-level evaluation may miss problems within a document: the correct file can be retrieved while the relevant section is absent from the returned chunks. A stronger evaluation tracks relevant passages, ranking quality, and whether the final answer is supported by the retrieved evidence.
The next step is to evaluate generation separately. Given the correct evidence, did the model produce a correct answer? Did it preserve the conditions and exceptions? Did it introduce claims that the sources do not support?
Separating retrieval and generation metrics makes failures easier to diagnose.
If retrieval recall is poor, investigate ingestion, chunking, indexing, query construction, and ranking. If retrieval is strong but answers remain incorrect, investigate context assembly, instructions, model behavior, and answer validation. Microsoft’s RAG Solution Design and Evaluation Guide discusses evaluation across the RAG workflow.
A retrieval system that worked well at launch can gradually become unreliable if its index no longer reflects the current source material.
This is not just an indexing problem. It is a lifecycle problem.
A production RAG system needs clear answers to questions such as:
How are changed documents detected?
How quickly are updated documents reindexed?
How are deleted or superseded documents removed?
How is the authoritative version identified?
How can a generated answer be traced back to its source?
How are retrieval regressions detected after an indexing or ranking change?
Document versioning and effective dates are especially important in enterprise systems. The newest document is not always the correct document for every question. A historical question may require the policy that was in force at a particular date.
Likewise, a document may be technically current but not applicable to a particular customer, product, or region.
Retrieval therefore needs to account for authority, time, scope, and access — not just semantic similarity.
An evaluation set should also evolve with the knowledge base. When important documents or business rules change, the team should update its test questions and verify that previously correct retrieval behavior still holds.
A more useful sequence is:
Verify the source material. Check whether the required information exists, is current, and can be extracted correctly.
Inspect the chunks. Confirm that headings, conditions, exceptions, table relationships, and relevant metadata survive preprocessing.
Examine retrieved results. Capture the actual passages returned for representative questions. Do not rely only on the final answer.
Improve candidate retrieval. Test keyword search, vector search, hybrid retrieval, and suitable filtering.
Evaluate ranking. Check whether relevant passages appear near the top and whether reranking improves their positions.
Validate evidence completeness. For multi-document questions, confirm that all necessary facts have been retrieved.
Evaluate generation independently. Give the model known-good evidence and assess whether its answer is accurate and supported.
Test changes against a fixed evaluation set. Compare results before and after each meaningful change to avoid relying on anecdotal improvements.
Monitor the lifecycle. Track document freshness, indexing failures, retrieval regressions, latency, and cost.
This sequence helps teams spend effort where it can make a measurable difference. It also prevents a common mistake: attempting to fix a retrieval failure with a more expensive model.
The system should be able to recognize that limitation.
If the retrieved documents do not establish whether a refund is permitted, the correct response may be to say that the available evidence is insufficient. If two authoritative sources conflict, the system may need to surface the conflict rather than silently select one. If a required document is inaccessible, the system should not invent its contents. This behavior requires more than a prompt that says “do not hallucinate.” It depends on how the application defines evidence sufficiency, handles conflicting sources, and validates answers.
For some questions, the application can require specific facts to be present before generation proceeds. For others, it can return a source-backed answer with clear limitations or ask the user for missing information. Abstention is not a substitute for good retrieval. It is a necessary part of handling the cases where retrieval cannot provide enough evidence.
A reliable system must know not only how to answer, but also when it does not have enough information to answer responsibly.
The Real Work Happens Before the Answer
RAG is often presented as a straightforward connection between a search index and a language model. That description hides much of the engineering work.
The quality of the final answer depends on the documents entering the system, how their content is represented, which passages are retrieved, how evidence is ranked and assembled, and whether the resulting context is sufficient for the question.
The model matters. But the model is only one component in a longer chain of decisions.
When an answer is wrong, start by examining what the model actually received. Was the right document available? Was the relevant passage preserved? Did retrieval find it? Did ranking place it high enough? Was the evidence complete, current, and authorized?
Those questions are often more useful than asking whether the model is smart enough.
The central question for a RAG system is not simply whether it can generate an answer. It is whether it retrieved the right evidence, from the right source, for the right question.
That is where reliable RAG engineering begins.