{"slug": "the-llm-is-not-the-hard-part-of-rag", "title": "The LLM Is Not the Hard Part of RAG", "summary": "An engineer argues that the hardest part of retrieval-augmented generation is retrieval quality and document structure, not the language model itself, illustrating how a RAG system can return fluent but unsupported answers when chunking splits a policy rule from its exception. The writeup walks through each pipeline stage—ingestion, chunking, indexing, retrieval, filtering, reranking and generation—and recommends chunk boundaries and metadata that preserve conditions, headings and cross-references so retrieved passages are complete enough to interpret.", "body_md": "Why retrieval quality, document structure, and relevance matter more than adding a model to your search system\n\nThe model produced a confident answer, but the relevant document was never retrieved.\n\nConsider a simple question: “Can a customer request a refund after the contract has been renewed?”\n\nImagine a knowledge base containing a general refund policy, a contract renewal procedure, and an older policy that has since been replaced. The system retrieves related information, passes it to the model, and generates an answer. But it misses the evidence that determines whether the answer is correct.\n\nThis is how a RAG system can fail without appearing broken. Search returns results. The model generates fluent text. The application responds successfully. Yet the answer is unsupported.\n\nRetrieval-augmented generation (RAG) is easy to explain: index documents, retrieve relevant passages, and give them to a language model. Building a reliable system is harder because finding text about a topic is not the same as finding the evidence needed to answer a question.\n\nThe difficult engineering work often happens before generation.\n\nA vector search might find a general refund policy while missing the renewal procedure that contains the decisive condition. The retrieved text may be semantically relevant without being sufficient to answer the question.\n\nThe model cannot reliably recover a rule it never sees. It may guess, combine unrelated conditions, or present an incomplete answer with unwarranted confidence.\n\nThis distinction changes how we debug the system. If retrieval is the failure point, changing the prompt or upgrading the model may increase cost without fixing the problem.\n\nA RAG pipeline is more than retrieval and generation.\n\nDocument Sources → Ingestion → Chunking → Indexing → Retrieval → Filtering → Reranking → LLM Generation → Answer\n\nEvery stage can introduce a different failure:\n\nDocument sources: Missing, duplicated, or outdated documents.\n\nIngestion: Content extracted incorrectly.\n\nChunking: Rules and exceptions separated from one another.\n\nIndexing: Missing or stale representations.\n\nRetrieval: Relevant evidence not found.\n\nFiltering: Useful or authorized evidence removed.\n\nReranking: Relevant passages ranked too low.\n\nGeneration: Evidence misinterpreted or unsupported claims added.\n\nFigure 1. A RAG pipeline. Answer quality depends on every stage, not only the language model.\n\nSuppose a refund policy states the general rule in one section and the renewal exception in another. A fixed-size split might separate the exception from the rule it modifies. Both chunks remain valid text, but neither may provide enough context when retrieved alone.\n\nVery small chunks can lose references and conditions. Large chunks preserve context but introduce unrelated content, increasing the amount of text the model must process.\n\nChunk boundaries should reflect the source material:\n\nPolicies need to preserve conditions, exceptions, and section headings.\n\nContracts may need clause boundaries and cross-references.\n\nProduct documentation benefits from keeping version details near feature descriptions.\n\nTables need their headings, units, and values to remain associated.\n\nMetadata such as document title, section, version, effective date, and source identifier can help preserve context without repeating entire documents in every chunk.\n\nConsider a table containing product limits. If the indexing pipeline extracts a value but loses the column heading, the retrieved text may contain the correct number without explaining what it means. The model then has to infer a relationship that the original document already defined.\n\nChunking is therefore part of knowledge modeling, not just text preprocessing. The goal is to create units that are focused enough to retrieve and complete enough to interpret.\n\n“Explain our refund policy” is different from “Does policy RF-204 apply after renewal?” The first may benefit from semantic matching; the second may depend on an exact identifier and a specific document version.\n\nHybrid retrieval combines keyword-based and vector search, then merges their results. Keyword search can capture critical terms such as “renewal” and exact identifiers, while vector search can tolerate differences in phrasing.\n\nThe value of hybrid search lies in combining these complementary strengths rather than assuming one retrieval method works equally well for every query.\n\nMicrosoft’s Hybrid Search Overview explains how text and vector search can work together.\n\nThe right retrieval strategy depends on the data and the questions users ask. A technical support system may need exact error-code matching. A policy assistant may depend more on semantic similarity, while still requiring exact policy identifiers and version constraints.\n\nThere is no reason to assume that one retrieval configuration is optimal for every knowledge domain.\n\nA passage can be highly similar to a question while omitting the condition that determines the outcome. Another passage with a lower score may contain the decisive exception.\n\nThis creates several engineering questions:\n\nHow many candidates should retrieval return?\n\nShould keyword and vector results be combined?\n\nWhich documents are authoritative?\n\nHow should document versions and effective dates affect ranking?\n\nWhen should a reranker reorder the candidates?\n\nWhat happens when the evidence is incomplete or contradictory?\n\nA typical approach is to retrieve a broader set of candidates and then rerank them using a more detailed relevance assessment. This can improve the order of results, but it cannot recover evidence that the initial retrieval stage never returned.\n\nFiltering also needs care. Removing obsolete or unauthorized documents is essential, but an overly broad filter can eliminate the only passage that answers the question.\n\nFor systems handling private or tenant-specific information, authorization must be enforced as part of retrieval. It should not depend on the language model choosing to ignore an unauthorized passage.\n\nMicrosoft’s RAG information retrieval guidance provides additional guidance on retrieval design.\n\nConsider: “Can this customer receive a refund under the current policy?”\n\nA reliable answer may require three pieces of evidence:\n\nThe current refund policy.\n\nThe customer’s contract renewal date.\n\nThe applicable exception for that product or contract type.\n\nRetrieving only the general policy is not enough. Retrieving the contract without the exception is not enough either.\n\nThis is where multi-step retrieval becomes relevant. The system may need to identify which facts are missing, retrieve additional documents, and combine the evidence before generating an answer.\n\nHowever, additional retrieval steps introduce their own risks. An early mistake can lead subsequent searches in the wrong direction. The system may also accumulate irrelevant passages or treat two documents as consistent when they are not.\n\nA useful design principle is to make evidence requirements explicit. For a question that depends on several conditions, define which facts must be established before an answer can be considered complete.\n\nThe system can then distinguish between a question that has sufficient evidence and one that still needs more information.\n\nSending an entire knowledge base into the prompt may increase latency, cost, and the amount of irrelevant information the model must process. Large inputs can also make it harder for a model to consistently use the most important details.\n\nMore context is not necessarily better context.\n\nA good retrieval system narrows the evidence to the material that matters for the current question. It preserves enough surrounding information to interpret that evidence correctly, while avoiding unrelated passages.\n\nThe objective is not to minimize context at all costs. It is to provide the smallest sufficient set of trustworthy evidence.\n\nThat balance depends on the task. A short troubleshooting question may need only one document section. A complex contract question may need multiple clauses and their cross-references.\n\nThe context window is a resource. Retrieval is how the system decides what deserves that resource.\n\nOne useful starting point is Recall@K: whether the expected relevant source appears among the top K retrieved results.\n\nIf a question has one expected source and that source appears in the top five results, its source recall at five is 1. If it does not appear, the score is 0. Across a set of questions, the average indicates how often retrieval found the expected evidence.\n\nFor questions that require multiple documents, the evaluation should also check whether all required sources were retrieved.\n\nConsider this small evaluation set:\n\nQuestionExpected sourceFound in top 5?RankWhich refund policy applies after renewal?Current refund policyTo be evaluated — Which policy applies to product XJ-420?Product policyTo be evaluated — What happens when a subscription is cancelled mid-cycle?Subscription policyTo be evaluated —\n\nThese are illustrative evaluation questions, not measured results. Each question should have a known set of relevant source documents. Track whether those sources appear in the top-K results and how highly they rank.\n\nA simple Python function can calculate source recall at K:\n\ndef retrieval_metrics_at_k(expected_sources, retrieved_sources, k=5):\n\n    \"\"\"\n\n    expected_sources: iterable of relevant source IDs\n\n    retrieved_sources: source IDs ordered by retrieval rank\n\n    \"\"\"\n\n    expected = set(expected_sources)\n\n    retrieved = retrieved_sources[:k]\n\n    if not expected:\n\n        return {\n\n            \"recall_at_k\": None,\n\n            \"all_sources_found\": None,\n\n        }\n\n    found = expected.intersection(retrieved)\n\n    return {\n\n        \"recall_at_k\": len(found) / len(expected),\n\n        \"all_sources_found\": found == expected,\n\n    }\n\nFor a single-source question, Recall@K is either 0 or 1. For a question requiring several sources, it measures the fraction of expected sources retrieved. The all_sources_found field indicates whether every expected source appeared in the top K.\n\nThis is only a starting point. Source-level evaluation may miss problems within a document: the correct file can be retrieved while the relevant section is absent from the returned chunks. A stronger evaluation tracks relevant passages, ranking quality, and whether the final answer is supported by the retrieved evidence.\n\nThe next step is to evaluate generation separately. Given the correct evidence, did the model produce a correct answer? Did it preserve the conditions and exceptions? Did it introduce claims that the sources do not support?\n\nSeparating retrieval and generation metrics makes failures easier to diagnose.\n\nIf retrieval recall is poor, investigate ingestion, chunking, indexing, query construction, and ranking. If retrieval is strong but answers remain incorrect, investigate context assembly, instructions, model behavior, and answer validation.\n\nMicrosoft’s RAG Solution Design and Evaluation Guide discusses evaluation across the RAG workflow.\n\nA retrieval system that worked well at launch can gradually become unreliable if its index no longer reflects the current source material.\n\nThis is not just an indexing problem. It is a lifecycle problem.\n\nA production RAG system needs clear answers to questions such as:\n\nHow are changed documents detected?\n\nHow quickly are updated documents reindexed?\n\nHow are deleted or superseded documents removed?\n\nHow is the authoritative version identified?\n\nHow can a generated answer be traced back to its source?\n\nHow are retrieval regressions detected after an indexing or ranking change?\n\nDocument versioning and effective dates are especially important in enterprise systems. The newest document is not always the correct document for every question. A historical question may require the policy that was in force at a particular date.\n\nLikewise, a document may be technically current but not applicable to a particular customer, product, or region.\n\nRetrieval therefore needs to account for authority, time, scope, and access — not just semantic similarity.\n\nAn evaluation set should also evolve with the knowledge base. When important documents or business rules change, the team should update its test questions and verify that previously correct retrieval behavior still holds.\n\nA more useful sequence is:\n\nVerify the source material. Check whether the required information exists, is current, and can be extracted correctly.\n\nInspect the chunks. Confirm that headings, conditions, exceptions, table relationships, and relevant metadata survive preprocessing.\n\nExamine retrieved results. Capture the actual passages returned for representative questions. Do not rely only on the final answer.\n\nImprove candidate retrieval. Test keyword search, vector search, hybrid retrieval, and suitable filtering.\n\nEvaluate ranking. Check whether relevant passages appear near the top and whether reranking improves their positions.\n\nValidate evidence completeness. For multi-document questions, confirm that all necessary facts have been retrieved.\n\nEvaluate generation independently. Give the model known-good evidence and assess whether its answer is accurate and supported.\n\nTest changes against a fixed evaluation set. Compare results before and after each meaningful change to avoid relying on anecdotal improvements.\n\nMonitor the lifecycle. Track document freshness, indexing failures, retrieval regressions, latency, and cost.\n\nThis sequence helps teams spend effort where it can make a measurable difference. It also prevents a common mistake: attempting to fix a retrieval failure with a more expensive model.\n\nThe system should be able to recognize that limitation.\n\nIf the retrieved documents do not establish whether a refund is permitted, the correct response may be to say that the available evidence is insufficient. If two authoritative sources conflict, the system may need to surface the conflict rather than silently select one. If a required document is inaccessible, the system should not invent its contents.\n\nThis behavior requires more than a prompt that says “do not hallucinate.” It depends on how the application defines evidence sufficiency, handles conflicting sources, and validates answers.\n\nFor some questions, the application can require specific facts to be present before generation proceeds. For others, it can return a source-backed answer with clear limitations or ask the user for missing information.\n\nAbstention is not a substitute for good retrieval. It is a necessary part of handling the cases where retrieval cannot provide enough evidence.\n\nA reliable system must know not only how to answer, but also when it does not have enough information to answer responsibly.\n\nThe Real Work Happens Before the Answer\n\nRAG is often presented as a straightforward connection between a search index and a language model. That description hides much of the engineering work.\n\nThe quality of the final answer depends on the documents entering the system, how their content is represented, which passages are retrieved, how evidence is ranked and assembled, and whether the resulting context is sufficient for the question.\n\nThe model matters. But the model is only one component in a longer chain of decisions.\n\nWhen an answer is wrong, start by examining what the model actually received. Was the right document available? Was the relevant passage preserved? Did retrieval find it? Did ranking place it high enough? Was the evidence complete, current, and authorized?\n\nThose questions are often more useful than asking whether the model is smart enough.\n\nThe central question for a RAG system is not simply whether it can generate an answer. It is whether it retrieved the right evidence, from the right source, for the right question.\n\nThat is where reliable RAG engineering begins.", "url": "https://wpnews.pro/news/the-llm-is-not-the-hard-part-of-rag", "canonical_source": "https://dev.to/ikilic/the-llm-is-not-the-hard-part-of-rag-dab", "published_at": "2026-10-17 16:58:44+00:00", "updated_at": "2026-10-17 17:03:51.833117+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "mlops"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-llm-is-not-the-hard-part-of-rag", "markdown": "https://wpnews.pro/news/the-llm-is-not-the-hard-part-of-rag.md", "text": "https://wpnews.pro/news/the-llm-is-not-the-hard-part-of-rag.txt", "jsonld": "https://wpnews.pro/news/the-llm-is-not-the-hard-part-of-rag.jsonld"}}