{"slug": "i-built-a-rag-assistant-then-found-out-my-architecture-change-made-it-worse", "title": "I built a RAG assistant, then found out my architecture change made it worse", "summary": "A developer built a hybrid retrieval-augmented generation (RAG) support assistant for a fictional B2B SaaS platform, 'Helix,' and found that hybrid search underperformed vector-only retrieval. Controlled before/after testing showed vector-only achieved 100% Hit@5 and 0.288 Doc-Precision@5, while hybrid (RRF) scored 94% and 0.248, respectively. The developer reported the negative result, emphasizing the importance of data-driven decisions over architectural sophistication.", "body_md": "I recently built a hybrid RAG (retrieval-augmented generation) support assistant for a fictional B2B SaaS platform, \"Helix,\" designed to answer customer-success questions grounded in a 100-document knowledge base of product docs, runbooks, and resolved support tickets. It cleared production-readiness evaluation thresholds comfortably: 0.939 faithfulness and 0.775 context precision on a 50-query RAGAs test set, against required floors of 0.70 and 0.60.\n\nBut the most useful thing that came out of the project wasn't the passing score. It was a hypothesis that turned out to be wrong, and what I did after finding that out.\n\nThe pipeline ingests a mixed-format 100-document corpus (Markdown product docs, PDF runbooks, HTML support tickets) into a Pinecone vector index, retrieves relevant context, and generates a grounded, citation-backed answer with an explicit confidence rating via an LCEL chain. Structured output is enforced with Pydantic (`answer`\n\n, `sources`\n\n, `confidence`\n\n), using `gpt-4o-mini`\n\nat `temperature=0`\n\n, because a support assistant answering the same question against the same context should give the same answer every time. Determinism mattered more than creative variation here.\n\nChunking wasn't one-size-fits-all. Three formats needed three strategies:\n\nResult: 95 of 100 documents ingested cleanly, 426 chunks produced.\n\nGoing in, I expected hybrid search (BM25 keyword matching combined with vector search via Reciprocal Rank Fusion) to outperform vector-only retrieval. The reasoning seemed solid: pure vector search can miss queries that hinge on an exact term (an error code, a specific field name), which get blurred by semantic embedding. BM25 should catch those, while vector search catches paraphrased matches BM25 would miss. Combining both should be strictly better.\n\nI measured it properly, running a controlled before/after comparison across all 50 test queries, tracking Hit@5 (whether the expected source document appeared in the top 5 results) and Doc-Precision@5.\n\n| Method | Hit@5 | Doc-Precision@5 |\n|---|---|---|\n| Vector-only | 100.0% (50/50) | 0.288 |\n| Hybrid (RRF) | 94.0% (47/50) | 0.248 |\n\nHybrid search didn't improve retrieval on this corpus. Vector-only was slightly better on both measures. My hypothesis was wrong.\n\nMy read: this corpus is mostly natural, well-formed prose, product docs and runbook text, rather than dense with the exact-match triggers (IDs, codes, rare tokens) where BM25 usually earns its keep. RRF fusion appears to occasionally push a correct vector hit that ranked just outside the top candidates out of the final top-5, in favor of a BM25 match that shares surface keywords but isn't the right document. That's a plausible explanation for the 3 queries hybrid lost that vector alone caught.\n\nThe end-to-end generation-level eval (context precision via RAGAs, judged at chunk-relevance rather than strict doc-ID match) still passed comfortably at 0.775. The two precision metrics aren't directly comparable, but it suggests hybrid isn't badly hurting overall answer quality even where it underperforms on the stricter doc-hit measure.\n\nI could have buried this, called hybrid a win because it's the more sophisticated-sounding architecture, and moved on. Instead I reported what the data actually said. If I ship this for real, that's the difference between a system that works and one that just looks like it does.\n\nDigging into the actual misses mattered more than the aggregate scores:\n\n**1. Pure retrieval miss.** A query about an HTTP 401 error, the exact kind of exact-term query hybrid was supposed to help with, still failed. The expected source docs were never retrieved. The system correctly said \"the provided context doesn't cover this\" rather than guessing (faithfulness scored 1.00 on the non-answer), but relevance and precision both scored 0.00. Root cause: retrieval, not generation.\n\n**2. Retrieval succeeded, generation still failed.** A webhook-configuration query correctly retrieved the right source document (context precision 1.00), but the model still said it couldn't answer. The specific chunk pulled from that file didn't happen to contain the actual instructions. Retrieving the right document doesn't guarantee retrieving the right chunk within it: a chunk-granularity gap, not a retrieval gap.\n\n**3. Incomplete synthesis on a hard, multi-document query.** A billing-dispute question needed four sources spanning billing docs, a refund runbook, and a resolved ticket. The system cited only one and produced a plausible but under-grounded answer (faithfulness 0.65). Top-k=5 didn't surface all four relevant documents at once for a query that genuinely needed cross-corpus synthesis.\n\nA fourth issue, not query-specific: citations occasionally leaked the literal string `\"doc_id: ...\"`\n\ninstead of a clean path, because the model copied the context block's label formatting. Fixed by rewording the prompt rule and changing the context label format.\n\nRanked by expected impact:\n\nIt would have been easy to write \"I built a hybrid RAG system, here are the great numbers\" and stop there. What I think actually demonstrates engineering judgment is the part where the more sophisticated architecture underperformed the simpler one, I noticed, I said so, and I could point to exactly which three queries broke and why. Faithfulness, context precision, and Hit@5 are useful summary numbers, but the failure analysis underneath them is where the real understanding of the system lives.\n\nRepo, full eval results, and code: [github.com/saftyy/hybrid-rag-enterprise-support](https://github.com/saftyy/hybrid-rag-enterprise-support)", "url": "https://wpnews.pro/news/i-built-a-rag-assistant-then-found-out-my-architecture-change-made-it-worse", "canonical_source": "https://dev.to/saftyy/i-built-a-rag-assistant-then-found-out-my-architecture-change-made-it-worse-1n7i", "published_at": "2026-08-14 18:50:02+00:00", "updated_at": "2026-08-14 19:05:24.976735+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "generative-ai", "ai-infrastructure"], "entities": ["Helix", "Pinecone", "RAGAs", "gpt-4o-mini", "LCEL", "Pydantic", "BM25", "Reciprocal Rank Fusion"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-rag-assistant-then-found-out-my-architecture-change-made-it-worse", "markdown": "https://wpnews.pro/news/i-built-a-rag-assistant-then-found-out-my-architecture-change-made-it-worse.md", "text": "https://wpnews.pro/news/i-built-a-rag-assistant-then-found-out-my-architecture-change-made-it-worse.txt", "jsonld": "https://wpnews.pro/news/i-built-a-rag-assistant-then-found-out-my-architecture-change-made-it-worse.jsonld"}}