Your baseline scored 0.000? Before you publish the win, here is the checklist I now run, because a zero from a baseline is almost never a result. It is usually your harness.
This week I watched a builder on r/Rag do something rare: he benchmarked his own memory library against plain RAG expecting a win, got three null results, and published the confounds instead of the victory. Two of his bugs are so common, and so quiet, that I think every RAG or agent-memory benchmark should gate against them by construction. This post is the checklist that came out of that thread.
His first run compared a memory arm retrieving k=20 sentence-level hits against a BM25 arm returning whole sessions. Same "top-k" on paper. In characters, one arm got 1.3k of context and the other got 11.9k. BM25 looked dramatically better.
Once he matched the budget, accuracy went 0.28 to 0.59 for the memory arms and the ranking flipped. The original result was a budget difference wearing a granularity costume.
The fix is embarrassingly simple: print the context size, in characters or tokens, next to every accuracy number, per arm. If a comparison does not state budget parity, the number does not mean anything. Chunked-vs-whole-document comparisons are especially prone to this because "top-k" hides a 10x budget difference in plain sight.
He ran a competing memory tool as a baseline. It scored 0.000. He re-ran it with a stronger extraction model. 0.000 again, logs clean. Very tempting to publish "competitor discards memories under load."
It was his bug, twice. His harness truncated each session to 6,000 characters before ingestion, cutting off the injected evidence. And he was passing limit=
to an API whose parameter is top_k=
, so his setting was silently ignored.
That second one deserves its own paragraph, because it is everywhere in this ecosystem: APIs that accept unknown kwargs without raising. They make a broken configuration look like a clean loss. In a benchmark, treat any client that swallows unknown parameters as hostile. Do not trust that a parameter did something. Assert it: if you move top_k
from 5 to 20, the retrieved count has to move. If it does not, your harness is configuring nothing. Out of that exchange came a harness structure I now consider the minimum for memory and RAG benchmarks. Three separate liveness gates, each with its own abort message, all pre-registered before the run.
1. A positive control per arm. Every arm gets a probe it cannot fail, built from the corpus's own ground truth. If any arm scores below a threshold you wrote down before the run, the whole benchmark aborts as "dead harness," not "clean loss." This is the gate that catches truncation bugs, auth failures, and empty stores. A zero is never evidence until this gate passes.
2. An evidence-in-context ceiling. For each probe, check whether the gold evidence is actually present in the retrieved context. Aggregated, this is the recall ceiling you report next to accuracy. In his broken run it was 3.5 percent. You cannot out-rank evidence that was never retrieved, and no reranker or prompt change will save a run whose ceiling is on the floor. Cheap to compute, needs no LLM calls.
3. A parameter-efficacy assertion. For each knob the benchmark claims to vary, assert that observable behavior changes when the knob moves. This is the gate that catches the swallowed-kwargs class of bug, and it is the one nobody runs.
The temptation is to fold these into a single sanity check. Do not. The whole value of a gate is that its failure names the broken layer. Evidence-in-context failing means retrieval never delivered. The positive control failing means the store or the answerer is dead. The efficacy assertion failing means your configuration is fiction. One merged gate tells you "something is wrong" and leaves the first hour of debugging still ahead of you.
There is a general lesson under all of this. A benchmark harness is software that lies by default, because every bug it has produces a plausible-looking number instead of a crash. The discipline that fixes it is the same one that fixes agent evaluation generally: never trust a claim the system makes about itself unless something the system cannot touch verifies it. A test that has never been proven to fail will happily pass forever.
Credit where it is due: the null-result writeup that prompted this is worth reading in full, including the part where he catches himself half-writing a false finding about a competitor before his own positive control saved him. Publishing that takes more spine than publishing a win.
I build a local-first assistant for the Mac, which is why I spend my days in retrieval evals. No product pitch here; the checklist stands on its own.