Same query.
Same documents.
Same model.
And the RAG eval can still hand back a different Recall@8.
Not because the model is flaky. Because of an ORDER BY
clause.
I didn't find this by watching a metric wobble. I found it reading the retrieval code, and realized the score would drift run to run even if the model never changed.
This came out of a habit I've adopted recently: I write the eval before the feature. Reviewing the retrieval pipeline behind my "Ask this Book" feature, I saw it: the retrieval layer wasn't deterministic.
My RAG implementation is intentionally simple: plain PostgreSQL and .NET. Two retrieval strategies over the same table:
The results are merged with Reciprocal Rank Fusion (RRF).
Here's the important part: RRF doesn't care about the retrieval scores. It only cares about rank.
If one retriever returns
A
B
C
instead of
B
A
C
RRF produces different fused scores. Different fused scores mean a different Top-K. Different Top-K means different Recall@K.
In RRF, order isn't a display detail. Order is data.
My lexical query ended like this:
ORDER BY score DESC
Looks perfectly reasonable. Except ts_rank_cd
produces ties surprisingly often. Multiple chunks can have exactly the same score.
And SQL only guarantees the ordering you explicitly request. If multiple rows compare equal, PostgreSQL is free to return them in any order.
Nothing changed. Same database. Same query. Same model. Only the order of equally-ranked rows. Yet that's enough for RRF to assign different ranks, producing different fused scores and a different evaluation result.
The semantic retrieval had the same issue. Distance ties are much rarer than lexical ties, but "rare" isn't good enough for an evaluation pipeline.
The fix was almost embarrassingly small.
Before:
ORDER BY score DESC
After:
ORDER BY score DESC, id
A deterministic tie-breaker on both retrieval queries. Now equal-scoring rows always appear in the same order, RRF receives the same input every run, and the Top-K stays identical.
Notice what didn't happen. The retrieval didn't become better. It became reproducible.
We spend a lot of effort making the model deterministic during evaluation: temperature 0, fixed datasets, golden answers, reproducible prompts.
But it's easy to assume everything underneath the model is already deterministic. Often it isn't. Retrieval. Ranking. Sampling. Data . Any non-deterministic stage in the pipeline can quietly invalidate your eval.
A fluctuating eval isn't just annoying. It's dangerous. Eventually you stop trusting the number, even when it's pointing at a real problem.
Before debugging the model, debug determinism. An evaluation can only be as deterministic as the pipeline feeding it. Same query. Same rows. Same order. Only then can you trust what your eval is telling you.
I build TextStack, an open-source reader for technical books, in .NET. This is from the retrieval layer behind its "Ask this Book" feature. Code on github.com/mrviduus/textstack.