{"slug": "the-rag-pipeline-i-wouldn-t-build-the-same-way-twice", "title": "The RAG Pipeline I Wouldn't Build the Same Way Twice", "summary": "A developer documented how their RAG architecture evolved away from default top-k vector search toward a system built on query-intent routing, hybrid BM25-plus-dense retrieval, and Reciprocal Rank Fusion. The engineer argues that semantic similarity is not the same as relevance, that fixed top-k retrieval and uniform chunk sizes break down on token-exact and multi-part queries, and that agentic workflows should only be adopted when their added complexity is justified.", "body_md": "**Subtitle:** A practical look at failure modes, query routing, hybrid retrieval, and knowing when agentic workflows are actually worth the complexity.\n\nEvery developer's first naive RAG system feels like magic. You chunk a few PDFs, pass them through an embedding model, save the vectors to a database, and hook up a top-k similarity search to an LLM. It takes 50 lines of Python, runs in a couple of seconds, and answers basic questions surprisingly well.\n\nThen real users show up.\n\nThey ask questions with precise technical identifiers (`error code 0x80070005`), temporal constraints (*\"What changed in our deployment policy last month?\"*), or broad multi-part requirements (*\"Compare our feature set with our competitor's pricing tier\"*).\n\nSuddenly, top-k vector similarity breaks down completely:\n\nThis article documents how my architectural thinking moved away from \"vector search by default\" toward a system built around query intent, multi-strategy retrieval, and selective complexity.\n\nA simple RAG architecture assumes a linear pipeline:\n\n`User Query -> Vector Search (Top-k) -> Context Assembly -> LLM Generation`\n\nThis model assumes that semantic similarity is equivalent to relevance. In practice, they are two very different metrics.\n\nDense vector embeddings condense the meaning of a text segment into a fixed-dimensional space (e.g., 768 or 1536 dimensions). This works well for conceptual queries (*\"How do I reset my credentials?\"*), but fails for token-exact queries (*\"What is the threshold for MAX_RETRY_ATTEMPTS in config.v2?\"*). \n\nBecause the vector space prioritizes overall semantic meaning, the specific token string `MAX_RETRY_ATTEMPTS` loses its distinctiveness.\n\nIf your chunks are too small (e.g., 200 tokens), you preserve fine-grained facts, but you lose the broader context needed to make sense of them. If your chunks are too large (e.g., 2000 tokens), you preserve context, but you dilute the relevance signal and waste the LLM's context window on noise.\n\nStatic top-k retrieval assumes that the ideal context size is constant. For a simple question, k=2 might be plenty. For a complex synthesis query, k=10 might still be insufficient. Retrieving a fixed number of chunks forces a trade-off between missing critical information and swamping the generation prompt with distraction.\n\nThe first major architectural shift was realizing that every incoming query does not deserve the same retrieval path. Treating all user input identically is an architectural flaw.\n\nInstead of passing every string directly to an embedding model, the system first passes the query through a fast classification layer—a lightweight query router.\n\nTo balance conceptual matching with token precision, hybrid retrieval combines sparse keyword search (BM25) with dense vector search.\n\nSparse models track exact word frequencies and term rarity, while dense models capture broader intent. Combining them ensures that exact product IDs or error codes aren't missed, while conceptual queries still return contextually relevant documents.\n\nOnce both retrievers return candidate lists, their score distributions must be normalized. A simple, effective technique for combining these distinct score metrics is **Reciprocal Rank Fusion (RRF)**:\n\n`RRF_Score(d) = Sum of (1 / (60 + Rank(d)))`\n\nWhere 60 is a constant that prevents high-ranking outliers from disproportionately dominating the output.\n\nRetrieval models prioritize recall—getting all potentially relevant chunks into a candidate list. Generative LLMs, on the other hand, require precision—receiving only the most useful context to formulate an answer.\n\nPassing 30 hybrid-retrieved candidate chunks directly into an LLM causes the **\"Lost in the Middle\"** phenomenon: transformer models pay disproportionate attention to information placed at the very beginning or the very end of their prompt context, often ignoring facts buried in the middle.\n\nBi-encoder models (standard vector embeddings) process the query and document chunks independently to create static vector representations. This is fast, but it prevents the query terms from interacting directly with the document terms.\n\nCross-encoder re-rankers process the query and document chunk **together** through transformer layers. This allows full cross-attention between every query token and every document token. While too computationally expensive to run against millions of database documents, running a cross-encoder against the top 20 or 30 retrieved candidates adds minimal latency while significantly sharpening relevance.\n\n*Engineering reliable RAG systems requires testing retrieval strategies against real-world query failure modes rather than relying solely on synthetic benchmarks.*\n\nWhen a static retrieval pipeline fails on multi-step reasoning, it's tempting to immediately rewrite the system as a fully autonomous agentic loop using frameworks like LangGraph or AutoGen.\n\nAn agentic loop gives the LLM tool access (e.g., query generation, external search, reflection) and allows it to run in a loop until it decides it has enough context to answer the user.\n\nHowever, adding agentic loops introduces significant engineering trade-offs:\n\nAn agentic approach is worth the complexity when:\n\nIf a query can be answered by routing it to a structured hybrid search path, adding an agentic framework is unnecessary overhead.\n\nA production-grade pipeline layout that balances latency, deterministic behavior, and retrieval precision follows this execution flow:\n\nIf I were rebuilding a RAG architecture from scratch today, these core engineering principles would guide my design decisions:\n\nIf you found this deep-dive useful, check out my earlier technical guides on building production AI systems:\n\n**Mithilesh Kumar** | AI Engineer & Full-Stack Developer\n\nMithilesh Kumar is an AI Engineer and Full-Stack Developer who specializes in building autonomous applications, Multi-Agent Systems, Agentic AI workflows, and Retrieval-Augmented Generation (RAG) pipelines. He is currently pursuing his Bachelor of Technology (B.Tech) in Computer Science & Engineering at Chandigarh Engineering College, Landran. He designs scalable architectures by blending AI-driven automation with modern web technologies, focusing on production reliability, retrieval precision, and practical engineering trade-offs.", "url": "https://wpnews.pro/news/the-rag-pipeline-i-wouldn-t-build-the-same-way-twice", "canonical_source": "https://dev.to/mithxcode/the-rag-pipeline-i-wouldnt-build-the-same-way-twice-3ga2", "published_at": "2026-09-18 16:01:00+00:00", "updated_at": "2026-09-18 16:23:12.867642+00:00", "lang": "en", "topics": ["ai-tools", "natural-language-processing", "large-language-models", "ai-agents", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/the-rag-pipeline-i-wouldn-t-build-the-same-way-twice", "markdown": "https://wpnews.pro/news/the-rag-pipeline-i-wouldn-t-build-the-same-way-twice.md", "text": "https://wpnews.pro/news/the-rag-pipeline-i-wouldn-t-build-the-same-way-twice.txt", "jsonld": "https://wpnews.pro/news/the-rag-pipeline-i-wouldn-t-build-the-same-way-twice.jsonld"}}