RAG, a zero usually means you've just benchmarked your own ability to write buggy code.
I've been seeing this everywhere. People run a "deep dive" into memory libraries, get a null result for the competitor, and call it a day. Here is the reality check for anyone attempting a practical tutorial on benchmarking or building a custom AI workflow.
The "Invisible" Budget Trap #
One of the most common delusions in RAG benchmarking is trusting "top-k." If one arm of your test retrieves 20 short sentences and the other retrieves 20 full documents, you aren't comparing retrieval quality—you're comparing context budgets. One model gets 1k tokens, the other gets 12k. Of course the one with more data wins.
The fix is painfully simple: log the actual character or token count next to every accuracy metric. If you aren't reporting budget parity, your numbers are essentially fiction.
The Silent API Fail #
The most annoying part of the current LLM ecosystem is APIs that swallow unknown keyword arguments without throwing an error. You pass limit=10
when the API actually expects top_k=10
, and the system just shrugs and uses the default. Your logs look clean, your code "runs," and your baseline scores a zero. You think the competitor is trash, but actually, your harness is just configuring nothing.
If you change a parameter and the retrieved count doesn't move, your harness is a lie.
The Trustworthiness Checklist #
If you want a real-world benchmark that doesn't make you look like an amateur, you need these three gates. If any of these fail, abort the run immediately. Positive Control: Every arm needs a probe itcannotfail based on ground truth. If the "easy" test scores zero, the harness is dead.Evidence-in-Context Ceiling: Calculate the recall ceiling. If the gold evidence isn't even in the retrieved context, no amount of prompt engineering or reranking will save the result.Parameter-Efficacy Assertion: Physically verify that turning a knob actually changes the output.
Stop treating benchmarks as a way to prove you're right and start treating them as a way to find where you're wrong. Most "breakthroughs" in agent memory are just people who forgot to check if their evidence was being truncated to 6,000 characters before ingestion.
Next Storyblok Pricing: A Complete Guide →