# The RAG Pipeline I Wouldn't Build the Same Way Twice > Source: > Published: 2026-09-18 16:01:00+00:00 **Subtitle:** A practical look at failure modes, query routing, hybrid retrieval, and knowing when agentic workflows are actually worth the complexity. Every developer's first naive RAG system feels like magic. You chunk a few PDFs, pass them through an embedding model, save the vectors to a database, and hook up a top-k similarity search to an LLM. It takes 50 lines of Python, runs in a couple of seconds, and answers basic questions surprisingly well. Then real users show up. They ask questions with precise technical identifiers (`error code 0x80070005`), temporal constraints (*"What changed in our deployment policy last month?"*), or broad multi-part requirements (*"Compare our feature set with our competitor's pricing tier"*). Suddenly, top-k vector similarity breaks down completely: This article documents how my architectural thinking moved away from "vector search by default" toward a system built around query intent, multi-strategy retrieval, and selective complexity. A simple RAG architecture assumes a linear pipeline: `User Query -> Vector Search (Top-k) -> Context Assembly -> LLM Generation` This model assumes that semantic similarity is equivalent to relevance. In practice, they are two very different metrics. Dense vector embeddings condense the meaning of a text segment into a fixed-dimensional space (e.g., 768 or 1536 dimensions). This works well for conceptual queries (*"How do I reset my credentials?"*), but fails for token-exact queries (*"What is the threshold for MAX_RETRY_ATTEMPTS in config.v2?"*). Because the vector space prioritizes overall semantic meaning, the specific token string `MAX_RETRY_ATTEMPTS` loses its distinctiveness. If your chunks are too small (e.g., 200 tokens), you preserve fine-grained facts, but you lose the broader context needed to make sense of them. If your chunks are too large (e.g., 2000 tokens), you preserve context, but you dilute the relevance signal and waste the LLM's context window on noise. Static top-k retrieval assumes that the ideal context size is constant. For a simple question, k=2 might be plenty. For a complex synthesis query, k=10 might still be insufficient. Retrieving a fixed number of chunks forces a trade-off between missing critical information and swamping the generation prompt with distraction. The first major architectural shift was realizing that every incoming query does not deserve the same retrieval path. Treating all user input identically is an architectural flaw. Instead of passing every string directly to an embedding model, the system first passes the query through a fast classification layer—a lightweight query router. To balance conceptual matching with token precision, hybrid retrieval combines sparse keyword search (BM25) with dense vector search. Sparse models track exact word frequencies and term rarity, while dense models capture broader intent. Combining them ensures that exact product IDs or error codes aren't missed, while conceptual queries still return contextually relevant documents. Once both retrievers return candidate lists, their score distributions must be normalized. A simple, effective technique for combining these distinct score metrics is **Reciprocal Rank Fusion (RRF)**: `RRF_Score(d) = Sum of (1 / (60 + Rank(d)))` Where 60 is a constant that prevents high-ranking outliers from disproportionately dominating the output. Retrieval models prioritize recall—getting all potentially relevant chunks into a candidate list. Generative LLMs, on the other hand, require precision—receiving only the most useful context to formulate an answer. Passing 30 hybrid-retrieved candidate chunks directly into an LLM causes the **"Lost in the Middle"** phenomenon: transformer models pay disproportionate attention to information placed at the very beginning or the very end of their prompt context, often ignoring facts buried in the middle. Bi-encoder models (standard vector embeddings) process the query and document chunks independently to create static vector representations. This is fast, but it prevents the query terms from interacting directly with the document terms. Cross-encoder re-rankers process the query and document chunk **together** through transformer layers. This allows full cross-attention between every query token and every document token. While too computationally expensive to run against millions of database documents, running a cross-encoder against the top 20 or 30 retrieved candidates adds minimal latency while significantly sharpening relevance. *Engineering reliable RAG systems requires testing retrieval strategies against real-world query failure modes rather than relying solely on synthetic benchmarks.* When a static retrieval pipeline fails on multi-step reasoning, it's tempting to immediately rewrite the system as a fully autonomous agentic loop using frameworks like LangGraph or AutoGen. An agentic loop gives the LLM tool access (e.g., query generation, external search, reflection) and allows it to run in a loop until it decides it has enough context to answer the user. However, adding agentic loops introduces significant engineering trade-offs: An agentic approach is worth the complexity when: If a query can be answered by routing it to a structured hybrid search path, adding an agentic framework is unnecessary overhead. A production-grade pipeline layout that balances latency, deterministic behavior, and retrieval precision follows this execution flow: If I were rebuilding a RAG architecture from scratch today, these core engineering principles would guide my design decisions: If you found this deep-dive useful, check out my earlier technical guides on building production AI systems: **Mithilesh Kumar** | AI Engineer & Full-Stack Developer Mithilesh Kumar is an AI Engineer and Full-Stack Developer who specializes in building autonomous applications, Multi-Agent Systems, Agentic AI workflows, and Retrieval-Augmented Generation (RAG) pipelines. He is currently pursuing his Bachelor of Technology (B.Tech) in Computer Science & Engineering at Chandigarh Engineering College, Landran. He designs scalable architectures by blending AI-driven automation with modern web technologies, focusing on production reliability, retrieval precision, and practical engineering trade-offs.