An 8-Point nDCG Bump That Costs Two Minutes Is Not Free

A new NVIDIA NeMo Retriever paper, "Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks," reports that an agentic retrieval setup lifts nDCG@10 by about 8.7 points over standard dense retrieval using the same embedding model, but at roughly 107.4 seconds of latency versus 0.67 seconds and about 764.1K input tokens plus 5.8K output tokens per query. A developer argues that retrieval evaluations reporting only quality metrics grade a different product than teams actually ship, and proposes an eval contract that reports latency distribution, token and dollar cost, and failure/fallback rate alongside nDCG.

A new NVIDIA NeMo Retriever paper, Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks https://arxiv.org/abs/2610.05750 arXiv 2610.05750, posted around 2026-10-05 , puts numbers on a trade everyone arguing about agentic RAG already feels. With the same embedding model, their agentic retrieval setup lifts nDCG@10 by about 8.7 points over standard dense retrieval. It also averages roughly 107.4 seconds of latency against about 0.67 seconds for the standard path, and burns on the order of 764.1K input tokens plus 5.8K output tokens per query. They also report competitive results on ViDoRe v3 and BRIGHT without task-specific specialization. Their retrieval-bench notes live in the public NeMo-Retriever https://github.com/NVIDIA/NeMo-Retriever/tree/main/retrieval-bench tree. Those are their numbers, not mine. I have not re-run the bench. What I want to formalize is the eval shape the paper forces into the open. Here is what I would require before I trusted an "agentic retrieval win," and what I would not claim yet. Teams will ship the wrong default. Most retrieval scoreboards still answer one question: "did we find better documents?" That question is real. It is incomplete. An 8-point nDCG bump that costs two minutes and three-quarters of a million input tokens is a different product than a sub-second dense call. If your dashboard only shows the quality column, the expensive path looks like a free upgrade. In production it is a budget, a timeout policy, and a fallback story. AI failures rarely fail loud. They fail as bills that looked like wins on the eval sheet. An eval contract that reports only retrieval quality without latency, token cost, and failure/fallback rate is grading a different product than the one you will ship. That is a preference with a limiter, not a benchmark result. The NeMo numbers stay theirs. What I am formalizing is the contract: quality, cost, and reliability sit in the same report, or you are not evaluating the system you will actually run. | Field | Question it answers | Failure if skipped | |---|---|---| | Retrieval quality | Did we find the right evidence under the stated metric? | Shipping on vibes | | Latency distribution | How long did the path take, including agent loops? | A "win" that times out in UX or SLA | | Token / dollar cost | What did one query cost on average and at the tail? | Quality gains that bankrupt the budget | | Failure / fallback rate | How often did the agentic path abort, stall, or hand off? | A green mean that hides a red path | | Same-model / same-corpus controls | Did we change one variable at a time? | Crediting the agent loop for an embedding upgrade | None of that needs a private harness. It needs someone to refuse a scoreboard that only prints nDCG. The paper's latency gap is the part I would steal first as a reporting habit. A sub-second dense path and a ~two-minute agentic path are not two settings of the same product. They are different products with different failure modes. One fails by missing evidence. The other can also fail by being too slow to use, too expensive to run at volume, or too brittle when a tool call hangs. If you only publish the quality delta, you are arguing about a system your users will never wait for. I would report latency and token cost next to the quality metric, not bury them in an appendix. The same goes for failure and fallback rate: every run that aborted or handed back to a cheaper path belongs in its own column. My public work is mostly retrieve-first systems, and the same split shows up there. A fluent answer can look like a retrieval win the same way a slow agent loop can look like a quality win. In both cases the fix is the same move: measure the retrieval half on its own, and refuse to let one number vouch for the whole product. The hybrid retrieval pipeline I keep public hybrid-rag-pipeline https://github.com/nomad-link-id/hybrid-rag-pipeline is built around that habit: measure retrieval separately before letting generation, or orchestration, speak for it. This week's paper is a cost-shaped version of the same lesson. Pick one retrieval path you already call "agentic" or "multi-hop" and add four things: Then ask whether the quality bump is still the headline once those columns are visible. An 8-point nDCG bump that costs two minutes is not free. Grade the product you will ship: quality, cost, and the rate at which the expensive path fails closed. Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com https://igoreduardo.com .