{"slug": "separating-retrieval-failures-from-generation-failures-in-a-rag-app", "title": "Separating retrieval failures from generation failures in a RAG app", "summary": "A team building a question-answering assistant over internal policy documents labeled about 200 failed conversations and found roughly 70% of failures were retrieval failures rather than generation errors, with many policy documents chunked mid-table so relevant rows were separated from their headers. After changing the chunking strategy, adding document titles to each chunk, and measuring retrieval hit rate separately from answer accuracy, answer quality improved without any model change. The team concluded that a single accuracy number hides where a pipeline problem lies, so measuring each stage separately avoids wasted effort.", "body_md": "A team built a question-answering assistant over internal policy documents. Users kept reporting wrong answers, and the team’s first instinct was to try a bigger model.\n\nBefore changing anything, they labeled about 200 failed conversations and sorted each into one of two groups: the right passage was never retrieved, or the right passage was retrieved but the model answered incorrectly. The split was roughly 70/30, with most failures coming from retrieval. Many policy documents had been chunked mid-table, so the relevant rows were separated from their headers.\n\nThey changed the chunking strategy, added document titles to each chunk, and measured retrieval hit rate separately from answer accuracy. Answer quality improved without any model change. The lesson is that a single “accuracy” number hides where the problem is, so measuring each stage of the pipeline separately saves a lot of wasted effort.", "url": "https://wpnews.pro/news/separating-retrieval-failures-from-generation-failures-in-a-rag-app", "canonical_source": "https://discuss.huggingface.co/t/separating-retrieval-failures-from-generation-failures-in-a-rag-app/191586#post_1", "published_at": "2026-11-09 04:40:48+00:00", "updated_at": "2026-11-09 04:52:00.713974+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "mlops"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/separating-retrieval-failures-from-generation-failures-in-a-rag-app", "markdown": "https://wpnews.pro/news/separating-retrieval-failures-from-generation-failures-in-a-rag-app.md", "text": "https://wpnews.pro/news/separating-retrieval-failures-from-generation-failures-in-a-rag-app.txt", "jsonld": "https://wpnews.pro/news/separating-retrieval-failures-from-generation-failures-in-a-rag-app.jsonld"}}