You set TOP_K=10
on your vector store. Ten candidate chunks means more signal for the model — that's the logic. Then you run npx ragscope
and the audit prints:
WARN 51/100 █████░░░░░ my-rag-service
│ "what are the pricing tiers?"
│
│ ✗ precision 30 ███░░░░░░░ 3/10 chunks used
│ ✗ efficiency 30 ███░░░░░░░ 70% tokens wasted
│ ✓ uniqueness 100 ██████████ chunks are distinct
│ ✓ coverage 100 ██████████ all chunks scored
│
│ → Reduce TOP_K 10→3 (only 3 chunks reached LLM)
│ → 70% of retrieved tokens never reached the LLM
│
Three of ten chunks made it into the final prompt. You paid for ten round-trips to the vector store and seven went nowhere. The other seven were fetched, scored, and discarded somewhere between your retrieval step and your prompt assembly code.
This is the gap ragscope (npm) was built to surface.
Retrieval and prompt assembly are separate steps, but most teams treat them as one. The gap is where the waste lives.
A typical RAG pipeline:
TOP_K
chunks from the vector storeStep 4 is where chunks disappear. Post-retrieval filtering — a score threshold, a hard token budget, a deduplication pass — silently drops chunks you already paid to retrieve. If your prompt assembly filters out anything below a certain confidence score and your vector store returns six chunks that don't clear it, those six fetches were wasted. The network round-trip still happened. The latency still accumulated.
The problem compounds over time. TOP_K=10
gets set as a safe default, the pipeline ships, and the setting never gets revisited. LLM eval scores look fine because the three chunks that do reach the prompt are the right ones. The waste is invisible in your evals — it only shows up in latency and cost.
Vector stores typically scale retrieval latency with TOP_K
. Fetching ten results takes measurably longer than fetching three, especially at tail latencies. When seven of those ten are discarded before the prompt, you're paying that latency premium on every query for nothing.
precision
ragscope runs as a local server (default port 4321) and receives spans from your pipeline via OTLP at http://localhost:4321/v1/traces
. No changes to your RAG code needed — configure your OpenTelemetry exporter to point there.
When both a retrieval span and an LLM span arrive for the same trace, ragscope compares them. The key field is inContext
on each RagChunk
. It inspects the full text of the LLM span's assembled prompt and checks whether each retrieved chunk's content appears in it — positionally, by string match. A chunk either appears in the prompt or it doesn't. No LLM, no heuristics, no sampling.
precision
starts as:
base = (chunks where inContext is true / total retrieved chunks) × 100
There's one additional penalty: if high-retrieval-rank chunks land in the middle of a long context window — the zone where LLM attention typically falls off — ragscope subtracts 12 points per buried chunk, capped at 36. This is the lost-in-the-middle detection. A pipeline where 3/10 chunks are used but two of those three are buried mid-context will score lower than one where the same 3 chunks sit at the prompt edges.
When the base falls below 60, ragscope generates a specific recommendation with the exact numbers: Reduce TOP_K 10→3 (only 3 chunks reached LLM)
. It prints this directly below the score bars, in the terminal, at the time the trace arrives — not in a cloud dashboard after a deploy.
efficiency
is a related metric: the fraction of retrieved tokens that reached the LLM. If precision
is 30 and your chunks are roughly uniform in size, efficiency
will also be around 30 — meaning 70% of the tokens you transferred from the vector store never appeared in a prompt. That shows up in retrieval latency and, depending on your pipeline, in processing time downstream.
Once you have precision
data, tuning is mechanical.
Follow the recommendation. ragscope prints Reduce TOP_K 10→3
because Math.max(used_chunks, 3)
gives you the minimum viable retrieval count with a floor of 3. Start there and re-run the audit against a few real queries.
Check efficiency alongside. Low
efficiency
paired with low precision
means your unused chunks are large — a chunking strategy problem as much as a TOP_K
problem. If efficiency
is high but precision
is low, your unused chunks are small and the token cost is modest; fixing TOP_K
will mostly recover the latency.Check uniqueness too. If the chunks that do reach your prompt include near-duplicates, the
uniqueness
metric will flag them. Two near-duplicate chunks in the prompt means one is redundant regardless of your TOP_K
setting — that's a deduplication-at-ingest problem, not a retrieval count problem. ragscope computes overlap between chunk pairs and surfaces high-overlap counts in the audit output.The typical path after a low-precision audit:
3/10 chunks used
— follow the Reduce TOP_K 10→3
recommendationprecision
should move above 75 (PASS)uniqueness
also flagged duplicates, fix chunking and re-runefficiency
is still low after fixing TOP_K
, chunks may be too large for the token budgetGoing from TOP_K=10
to TOP_K=3
on a pipeline that was only ever using 3 chunks means 70% fewer vector store round-trips on every query. No model changes, no prompt rewriting, no reranker added.
The assumption that more retrieval means better answers is almost never validated against what the model actually sees. TOP_K
defaults get set once and forgotten. Eval scores stay flat because the chunks that do reach the LLM are the right ones — the waste doesn't affect quality, just cost and latency.
npx ragscope
gives you precision
, efficiency
, and uniqueness
in the time it takes to run a dev command. If you haven't checked what fraction of your retrieved chunks survive to the prompt, that's the number to look at first.
precision
measures what fraction of retrieved chunks appear in the final LLM prompt — anything below 60/100 triggers a Reduce TOP_K
recommendation with exact numbersTOP_K
; fetching chunks you discard is pure overhead with no quality benefitefficiency
paired with low precision
points to a chunking strategy problem, not just a TOP_K
problemuniqueness
means near-duplicate chunks in the prompt — fix at ingest, not by adjusting retrieval count