Stop Paying For Retrieval Latency On Chunks You Never Use In The Prompt

A developer built RagScope, an open-source tool that audits RAG pipelines to reveal wasted retrieval latency from chunks fetched but never used in the final prompt. The tool uses OpenTelemetry traces to compare retrieved chunks against the actual prompt text, surfacing metrics like precision and efficiency. In one example, a pipeline with TOP_K=10 used only 3 chunks, wasting 70% of retrieved tokens.

You set TOP K=10 on your vector store. Ten candidate chunks means more signal for the model — that's the logic. Then you run npx ragscope and the audit prints: WARN 51/100 █████░░░░░ my-rag-service │ "what are the pricing tiers?" │ │ ✗ precision 30 ███░░░░░░░ 3/10 chunks used │ ✗ efficiency 30 ███░░░░░░░ 70% tokens wasted │ ✓ uniqueness 100 ██████████ chunks are distinct │ ✓ coverage 100 ██████████ all chunks scored │ │ → Reduce TOP K 10→3 only 3 chunks reached LLM │ → 70% of retrieved tokens never reached the LLM │ Three of ten chunks made it into the final prompt. You paid for ten round-trips to the vector store and seven went nowhere. The other seven were fetched, scored, and discarded somewhere between your retrieval step and your prompt assembly code. This is the gap ragscope https://github.com/Sidd27/ragscope npm https://www.npmjs.com/package/ragscope was built to surface. Retrieval and prompt assembly are separate steps, but most teams treat them as one. The gap is where the waste lives. A typical RAG pipeline: TOP K chunks from the vector storeStep 4 is where chunks disappear. Post-retrieval filtering — a score threshold, a hard token budget, a deduplication pass — silently drops chunks you already paid to retrieve. If your prompt assembly filters out anything below a certain confidence score and your vector store returns six chunks that don't clear it, those six fetches were wasted. The network round-trip still happened. The latency still accumulated. The problem compounds over time. TOP K=10 gets set as a safe default, the pipeline ships, and the setting never gets revisited. LLM eval scores look fine because the three chunks that do reach the prompt are the right ones. The waste is invisible in your evals — it only shows up in latency and cost. Vector stores typically scale retrieval latency with TOP K . Fetching ten results takes measurably longer than fetching three, especially at tail latencies. When seven of those ten are discarded before the prompt, you're paying that latency premium on every query for nothing. precision ragscope runs as a local server default port 4321 and receives spans from your pipeline via OTLP at http://localhost:4321/v1/traces . No changes to your RAG code needed — configure your OpenTelemetry exporter to point there. When both a retrieval span and an LLM span arrive for the same trace, ragscope compares them. The key field is inContext on each RagChunk . It inspects the full text of the LLM span's assembled prompt and checks whether each retrieved chunk's content appears in it — positionally, by string match. A chunk either appears in the prompt or it doesn't. No LLM, no heuristics, no sampling. precision starts as: base = chunks where inContext is true / total retrieved chunks × 100 There's one additional penalty: if high-retrieval-rank chunks land in the middle of a long context window — the zone where LLM attention typically falls off — ragscope subtracts 12 points per buried chunk, capped at 36. This is the lost-in-the-middle detection. A pipeline where 3/10 chunks are used but two of those three are buried mid-context will score lower than one where the same 3 chunks sit at the prompt edges. When the base falls below 60, ragscope generates a specific recommendation with the exact numbers: Reduce TOP K 10→3 only 3 chunks reached LLM . It prints this directly below the score bars, in the terminal, at the time the trace arrives — not in a cloud dashboard after a deploy. efficiency is a related metric: the fraction of retrieved tokens that reached the LLM. If precision is 30 and your chunks are roughly uniform in size, efficiency will also be around 30 — meaning 70% of the tokens you transferred from the vector store never appeared in a prompt. That shows up in retrieval latency and, depending on your pipeline, in processing time downstream. Once you have precision data, tuning is mechanical. Follow the recommendation. ragscope prints Reduce TOP K 10→3 because Math.max used chunks, 3 gives you the minimum viable retrieval count with a floor of 3. Start there and re-run the audit against a few real queries. Check efficiency alongside. Low efficiency paired with low precision means your unused chunks are large — a chunking strategy problem as much as a TOP K problem. If efficiency is high but precision is low, your unused chunks are small and the token cost is modest; fixing TOP K will mostly recover the latency. Check uniqueness too. If the chunks that do reach your prompt include near-duplicates, the uniqueness metric will flag them. Two near-duplicate chunks in the prompt means one is redundant regardless of your TOP K setting — that's a deduplication-at-ingest problem, not a retrieval count problem. ragscope computes overlap between chunk pairs and surfaces high-overlap counts in the audit output.The typical path after a low-precision audit: 3/10 chunks used — follow the Reduce TOP K 10→3 recommendation precision should move above 75 PASS uniqueness also flagged duplicates, fix chunking and re-run efficiency is still low after fixing TOP K , chunks may be too large for the token budgetGoing from TOP K=10 to TOP K=3 on a pipeline that was only ever using 3 chunks means 70% fewer vector store round-trips on every query. No model changes, no prompt rewriting, no reranker added. The assumption that more retrieval means better answers is almost never validated against what the model actually sees. TOP K defaults get set once and forgotten. Eval scores stay flat because the chunks that do reach the LLM are the right ones — the waste doesn't affect quality, just cost and latency. npx ragscope gives you precision , efficiency , and uniqueness in the time it takes to run a dev command. If you haven't checked what fraction of your retrieved chunks survive to the prompt, that's the number to look at first. precision measures what fraction of retrieved chunks appear in the final LLM prompt — anything below 60/100 triggers a Reduce TOP K recommendation with exact numbers TOP K ; fetching chunks you discard is pure overhead with no quality benefit efficiency paired with low precision points to a chunking strategy problem, not just a TOP K problem uniqueness means near-duplicate chunks in the prompt — fix at ingest, not by adjusting retrieval count