10 RAG Pipeline Mistakes that Quietly Kill Retrieval Quality

A support engineer's RAG chatbot confidently provided the wrong refund policy because the retriever pulled a paragraph mentioning 'refund' four times from a discontinued product line, illustrating that most RAG failures stem from retrieval errors rather than generation. The article outlines ten specific mistakes that degrade retrieval quality in production RAG pipelines, from bad chunking to missing reranking, with concrete fixes and metrics to measure.

Member-only story 10 RAG Pipeline Mistakes that Quietly Kill Retrieval Quality Ten specific RAG retrieval mistakes, from bad chunking to missing reranking, with concrete fixes, code, and what to measure to know it A support engineer once showed me a RAG chatbot that explained, in beautiful prose, the wrong refund policy. The retriever had pulled a paragraph that mentioned “refund” four times. Topically close. Also for a discontinued product line. The LLM did exactly what it was told: answer confidently based on the context it was given. Nobody had told it the context was wrong. That’s the pattern behind most RAG failures I’ve watched get debugged. The team spends two weeks swapping the LLM, rewriting the system prompt, adding “please only answer from context” in bold, and the hallucination rate barely moves. Because the problem was never generation. It was retrieval. The model was reasoning correctly over evidence that was already wrong before it arrived. This piece walks through ten concrete mistakes that degrade retrieval quality in production RAG pipelines, in the order they usually get discovered: chunking errors first, since they’re upstream of everything, then retrieval architecture, then reranking and embedding choice, then the evaluation gap that lets all of the above ship…