Finding the Right Answers from Thousands of Documents: A Smarter RAG Approach

A production-ready multi-stage RAG pipeline that combines vector search with BM25 keyword retrieval and Reciprocal Rank Fusion can scale to thousands of documents while improving answer quality, according to a technical guide. The approach retrieves a wider set of candidates, reranks them precisely, and reduces token consumption by sending only the most relevant chunks to the LLM. The pipeline uses ChromaDB for vector storage and the all-MiniLM-L6-v2 embedding model.

RAG is often presented as a simple, three-step architecture: put documents into a vector database, convert the user’s question into an embedding, retrieve a handful of chunks, and hand them to an LLM. That approach is a great proof of concept. It is also where most RAG projects quietly stall. But what happens when the knowledge base grows to hundreds or thousands of documents? Retrieval becomes more challenging, irrelevant chunks can reach the LLM, token consumption increases, and the quality of the final answer becomes increasingly dependent on retrieval quality. This article shares information on building a production-ready, multi-stage RAG pipeline that can scale without simply sending more and more context to the LLM. Retrieval-Augmented Generation allows an AI model to answer questions using external knowledge. Instead of relying only on the LLM’s internal knowledge, the system retrieves relevant information from a knowledge base and provides it to the model as context before generating an answer. User Question → Retrieve Relevant Context → LLM → Answer The important word here is relevant . A powerful LLM cannot consistently produce high-quality answers if the retrieval system provides incomplete or irrelevant context. A typical RAG implementation looks like this Documents → Chunk Documents → Create Embeddings → Store in Vector DB And on the query side: User Question → Create Query Embedding → Vector Search → Retrieve Chunks → Send to LLM → Generate Answer For a small dataset, this approach may work perfectly well. However, as the knowledge base grows, several challenges start to appear. Scaling With thousands of documents, there may be hundreds of thousands of chunks. A vector search may return content that is semantically similar but does not actually answer the user’s question. Response Time One common solution is to retrieve more chunks. But more chunks mean more processing and potentially more context sent to the LLM. Token Consumption Not every retrieved chunk is useful. Passing 30 or 50 chunks directly to the LLM can significantly increase token consumption while adding unnecessary noise. Response Quality Semantic similarity does not always mean answer relevance. The overall philosophy is: Retrieve broadly. Combine intelligently. Rerank precisely. Then let the LLM reason. The first stage focuses on recall . The objective is not necessarily to find the perfect chunks immediately. Instead, the objective is to identify a wider set of potentially relevant candidates. Documents are split into chunks and converted into embeddings. These embeddings are stored in a vector database such as ChromaDB. When a user submits a question, the query is converted into an embedding using a model such as: all-MiniLM-L6-v2 This model produces vector representation of the text. The vector database then searches for semantically similar chunks. For example: User Query: “How do I rotate secrets?” The important idea is to retrieve a wider net of candidates . Vector search is fast and excellent at identifying semantic similarity, making it an ideal first stage. But semantic search should not be the only retrieval mechanism. Vector search is good at understanding semantic meaning, but it may miss exact keywords, error codes, commands, or technical terms. BM25 complements vector search by performing keyword-based retrieval. Instead of choosing one approach, the results from both searches are combined using Reciprocal Rank Fusion RRF . RRF uses the ranking position of each result rather than directly comparing scores from different retrieval methods. Vector Search + BM25 → RRF Fusion → Better Candidate Ranking This creates a hybrid retrieval mechanism that combines semantic understanding with exact keyword matching. After the first two stages, the pipeline may have reduced thousands of chunks to perhaps 20 or 30 strong candidates. The next question is: Which of these chunks actually answers the user’s question? This is where a cross-encoder becomes useful. An embedding model processes the query and document separately and compares their vector representations. A cross-encoder processes them together: Query + Candidate Chunk → Cross-Encoder → Relevance Score For example: Query: “How do I rotate Cloud secrets?” Candidate Scores: 0.98 Cloud Secret Manager supports automatic rotation…0.81 Secret lifecycle defines credential management…0.22 Cloud provides several storage services…0.07 Metadata helps organize enterprise data… The cross-encoder can make a much more precise relevance decision because it sees the query and candidate chunk together. The trade-off is performance. A cross-encoder is slower than vector search, so running it against thousands of chunks would be inefficient. That is why the earlier stages are important: 100,000 Chunks → Vector + BM25 Retrieval → 30 Candidates → Cross-Encoder → Top 5 This follows a simple principle: Use fast retrieval to reduce the search space, then use more precise models on a smaller candidate set . Only after the retrieval and ranking stages do we send context to the LLM. The final top-ranked chunks are assembled into a context prompt and passed to Mistral, or any other LLM. Top Relevant Chunks → Context Builder → Mistral / LLM → Final Answer A simple prompt could instruct the model to: Answer using only the provided context. Avoid making unsupported claims. Clearly state when the answer cannot be found. Provide source information where possible. The LLM can now focus on what it does best: reasoning, connecting information, summarizing, and generating a clear response. It does not need to search through 50 loosely related chunks. Imagine a knowledge base containing: 2,000 Documents - 100,000 Chunks A user asks: How can I troubleshoot a failed secret rotation? The pipeline could work like this: Every stage has a specific purpose. Instead of asking one component to do everything, the pipeline allows each component to do what it is best at. A RAG pipeline should be tuned based on the type and size of the knowledge base. A few useful practices include: The goal is not to find a single perfect configuration, but to continuously tune the pipeline based on measurable results. Testing is one of the most overlooked parts of a RAG project. A common approach is to ask a few users to test the system manually. They may submit a handful of questions and review the responses. This is useful, but it does not provide enough coverage for a knowledge base containing thousands of documents. The questions may be too simple, predictable, or focused on only a small part of the dataset. A better approach is to build an AI-based RAG test agent . The test agent can: • Read the knowledge base. • Generate a large set of questions. • Categorize questions by difficulty. • Create expected answers. • Identify expected source documents or chunks. • Execute the questions against the RAG pipeline. • Evaluate the responses. The architecture could look like this: This creates a continuous testing cycle and provides measurable insight into how well the RAG pipeline performs before moving it to production. Building a basic RAG system is relatively easy. Building a reliable RAG system that works well at scale is a different challenge. A multi-stage pipeline reduces a large knowledge base into a small, high-quality context. Broad Retrieval → Hybrid Search → RRF Fusion → Reranking → Quality context → LLM Resoning The key is to let each component do what it does best. Testing is equally important. Rather than relying only on a small number of manually created questions, an AI-based test agent can generate a much broader evaluation dataset and continuously measure retrieval and answer quality. The goal is not to make the RAG architecture unnecessarily complex. The goal is to make sure that when the LLM finally receives context, it receives the right information Because in a RAG system, answer quality often depends on the context passed to the LLM. Retrieve broadly. Rank intelligently. Rerank precisely. Then let the AI reason. Finding the Right Answers from Thousands of Documents: A Smarter RAG Approach https://pub.towardsai.net/finding-the-right-answers-from-thousands-of-documents-a-smarter-rag-approach-af2c59f9a8dd was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.