{"slug": "your-rag-eval-isn-t-flaky-your-retrieval-is-non-deterministic", "title": "Your RAG Eval Isn't Flaky. Your Retrieval Is Non-Deterministic.", "summary": "A developer discovered that non-deterministic retrieval ordering caused fluctuating RAG evaluation results. The issue was traced to SQL ORDER BY clauses that allowed ties, which changed the input to Reciprocal Rank Fusion and produced different Top-K results. Adding a deterministic tie-breaker (ORDER BY score DESC, id) fixed the reproducibility problem without improving retrieval quality.", "body_md": "Same query.\n\nSame documents.\n\nSame model.\n\nAnd the RAG eval can still hand back a different Recall@8.\n\nNot because the model is flaky. Because of an `ORDER BY`\n\nclause.\n\nI didn't find this by watching a metric wobble. I found it reading the retrieval code, and realized the score would drift run to run even if the model never changed.\n\nThis came out of a habit I've adopted recently: I write the eval before the feature. Reviewing the retrieval pipeline behind my \"Ask this Book\" feature, I saw it: the retrieval layer wasn't deterministic.\n\nMy RAG implementation is intentionally simple: plain PostgreSQL and .NET. Two retrieval strategies over the same table:\n\nThe results are merged with Reciprocal Rank Fusion (RRF).\n\nHere's the important part: **RRF doesn't care about the retrieval scores. It only cares about rank.**\n\nIf one retriever returns\n\n```\nA\nB\nC\n```\n\ninstead of\n\n```\nB\nA\nC\n```\n\nRRF produces different fused scores. Different fused scores mean a different Top-K. Different Top-K means different Recall@K.\n\nIn RRF, order isn't a display detail. Order is data.\n\nMy lexical query ended like this:\n\n```\nORDER BY score DESC\n```\n\nLooks perfectly reasonable. Except `ts_rank_cd`\n\nproduces ties surprisingly often. Multiple chunks can have exactly the same score.\n\nAnd SQL only guarantees the ordering you explicitly request. If multiple rows compare equal, PostgreSQL is free to return them in any order.\n\nNothing changed. Same database. Same query. Same model. Only the order of equally-ranked rows. Yet that's enough for RRF to assign different ranks, producing different fused scores and a different evaluation result.\n\nThe semantic retrieval had the same issue. Distance ties are much rarer than lexical ties, but \"rare\" isn't good enough for an evaluation pipeline.\n\nThe fix was almost embarrassingly small.\n\nBefore:\n\n```\nORDER BY score DESC\n```\n\nAfter:\n\n```\nORDER BY score DESC, id\n```\n\nA deterministic tie-breaker on both retrieval queries. Now equal-scoring rows always appear in the same order, RRF receives the same input every run, and the Top-K stays identical.\n\nNotice what didn't happen. The retrieval didn't become better. **It became reproducible.**\n\nWe spend a lot of effort making the model deterministic during evaluation: temperature 0, fixed datasets, golden answers, reproducible prompts.\n\nBut it's easy to assume everything underneath the model is already deterministic. Often it isn't. Retrieval. Ranking. Sampling. Data loading. Any non-deterministic stage in the pipeline can quietly invalidate your eval.\n\nA fluctuating eval isn't just annoying. It's dangerous. Eventually you stop trusting the number, even when it's pointing at a real problem.\n\nBefore debugging the model, debug determinism. An evaluation can only be as deterministic as the pipeline feeding it. Same query. Same rows. Same order. Only then can you trust what your eval is telling you.\n\n*I build TextStack, an open-source reader for technical books, in .NET. This is from the retrieval layer behind its \"Ask this Book\" feature. Code on github.com/mrviduus/textstack.*", "url": "https://wpnews.pro/news/your-rag-eval-isn-t-flaky-your-retrieval-is-non-deterministic", "canonical_source": "https://dev.to/mrviduus/your-rag-eval-isnt-flaky-your-retrieval-is-non-deterministic-42ab", "published_at": "2026-07-14 14:48:00+00:00", "updated_at": "2026-07-14 14:59:15.644610+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["TextStack", "PostgreSQL", ".NET", "RRF", "ts_rank_cd"], "alternates": {"html": "https://wpnews.pro/news/your-rag-eval-isn-t-flaky-your-retrieval-is-non-deterministic", "markdown": "https://wpnews.pro/news/your-rag-eval-isn-t-flaky-your-retrieval-is-non-deterministic.md", "text": "https://wpnews.pro/news/your-rag-eval-isn-t-flaky-your-retrieval-is-non-deterministic.txt", "jsonld": "https://wpnews.pro/news/your-rag-eval-isn-t-flaky-your-retrieval-is-non-deterministic.jsonld"}}