For a while, my retrieval setting was top_k = 10
. It felt responsible. If I only pulled back 3 chunks and the right one wasn't among them, the answer would be wrong for a reason that had nothing to do with the model. Pulling back 10 felt like insurance — cast a wide net, let the LLM sort out what actually mattered.
The bot's answers didn't get better. Some of them got worse.
I remember one question in particular — something specific, with a clear correct chunk sitting somewhere in the middle of those 10. The LLM's answer wandered. It grabbed a detail from a chunk that was only loosely related, ignored the one that actually answered the question, and produced something that read confidently but missed the point. The right information was in the prompt. It just wasn't the information the model reached for.
That's when I went back and actually read what I'd been sending. Ten chunks, most of them tangential, all competing for the model's attention in one long prompt. The correct answer wasn't easy to find in there — even for me, reading it slowly, on purpose. I'd been treating "send more context" as a safety net. It was actually working against the model, and against my bill, on every single request.
So I tried the opposite: retrieve more candidates than I needed, but add a reranking step that scores them for actual relevance, and only send the LLM the top 3 after that. Fewer tokens per prompt. And, somewhat counter to what I expected, better answers — not despite sending less, but because of it.
That flipped something I'd assumed without ever testing it: more retrieved context isn't more safety. Past a point, it's noise the model has to wade through, paid for at input-token prices, on every question.
Retrieval quality determines how much you end up overpaying the LLM to compensate for it. Sending more chunks isn't a safety margin — it's a tax that also makes the model's job harder.
Every earlier episode in this series was about getting clean material into the vector database. This one is about the last decision before that material reaches the model: how much of it, and how well-chosen, actually makes it into the prompt. Get that wrong, and no amount of extraction quality, chunking care, or metadata precision from Episodes 2 through 5 will save you — the right answer can be sitting in the prompt and still get missed.
The most direct cost is the one from the story: every extra chunk in top_k
is extra input tokens, paid on every single query.
top_k = 3
│
▼
Fewer tokens in the prompt, cheaper per query
top_k = 10
│
▼
More tokens in the prompt, more expensive per query
│
▼
...and most of those extra chunks are redundant or irrelevant
This part is intuitive once you say it out loud — more chunks costs more tokens. What's easy to miss is that this cost buys you very little if the extra chunks aren't actually useful. You're not paying for more safety. You're paying for more text the model has to read past.
This is the cost that surprised me in the story, and it doesn't show up as a token count — it shows up as answer quality.
Language models don't weigh every part of a long prompt equally. Information buried in the middle of a long, noisy context is measurably easier for a model to underweight or miss than information near the start or end of a tightly-focused prompt. This is often described as the "lost in the middle" effect, and it means a longer prompt isn't just more expensive — it can be actively worse at surfacing the right answer, even when the right chunk is technically present.
Long, noisy prompt (top_k = 10)
│
▼
Correct chunk is in there, but surrounded by 9 distractors
│
▼
Model's attention gets diluted across all of them
│
▼
Answer misses or underweights the correct chunk
│
▼
Wrong or incomplete answer, despite having the right information
This is the cost that turns "safety margin" logic on its head. More context isn't neutral-at-worst. Past a point, it can make the model's job harder while also making you pay more for the privilege.
If sending fewer, better chunks is the goal, you need a way to know which chunks are actually the best ones — vector similarity alone isn't always precise enough to trust for the final cut.
A reranker sits between retrieval and generation: retrieve a wider set of candidates cheaply, score them for relevance with a more precise (and more expensive per-item) model, then send only the top few to the LLM.
Retrieve 20-30 candidates (cheap, wide net)
│
▼
Rerank them for actual relevance (adds compute + latency, but small per item)
│
▼
Send only the top 3 to the LLM (expensive-per-token stage)
│
▼
Net effect: extra cost at a cheap stage, savings at the expensive stage
This is the same shape as every quality-vs-cost decision earlier in this series — you're moving cost to an earlier, cheaper stage to avoid paying for it at a later, more expensive one. Reranking isn't free. But it's usually cheap relative to the LLM tokens it saves, because scoring candidate relevance is a fundamentally lighter task than generating a full answer.
This is the cost I think is easiest to fall into without noticing, because it looks like progress instead of a symptom.
When retrieval quality is poor, one common instinct is to reach for a bigger, more capable model — reasoning that a smarter model can compensate for messier context. Sometimes it can, partially. But it's an expensive way to patch a cheaper problem.
Retrieval sends noisy, poorly-chosen context
│
▼
Answers are inconsistent
│
▼
Team upgrades to a larger, more expensive model to compensate
│
▼
Cost per query goes up significantly
│
▼
Underlying retrieval problem is still there, just harder to see
A bigger model paying attention to bad context is still paying attention to bad context — it's just more expensive while doing it. The fix that actually addresses the problem (better retrieval, reranking, tighter top_k) usually costs far less than the fix that just papers over it (a bigger model). This is the retrieval-episode version of Episode 2's lesson: fixing quality upstream is almost always cheaper than compensating for it downstream.
Same pattern as the earlier episodes — a small worked example to make the trade-off tangible. These figures are illustrative, not measured from a real system.
Example
top_k = 10, no reranking
│
▼
~10 chunks × ~300 tokens each ≈ 3,000 prompt tokens
│
▼
Correct chunk present, but diluted among 9 others
versus
top_k = 30 candidates → reranked → top 3 sent to LLM
│
▼
~3 chunks × ~300 tokens each ≈ 900 prompt tokens
│
▼
Correct chunk present, and it's one of only 3 — much harder to miss
Roughly a 3x drop in prompt tokens on every single query, plus a real shot at fewer wrong-answer retries, for the cost of one lightweight reranking step that's cheap relative to what it's replacing. This is the same shape of math as Episode 3's chunk-size example — a small change earlier in the pipeline, multiplied by every request that follows it.
More context was never the safety net it felt like. Past a certain point, every extra chunk you retrieve is a small, compounding tax — paid in tokens, paid in the model's attention, and sometimes paid twice over when a team responds to bad retrieval by reaching for a bigger model instead of a better one.
The best retrieval system isn't the one that sends the most. It's the one that's confident enough to send less.
I started this series certain that embeddings were where RAG became expensive. Six episodes later, the actual pattern looks nothing like that:
Extraction → decides what knowledge survives at all
Chunking → decides how that knowledge gets sliced, and how much it costs to slice it
Metadata → decides which slices retrieval is even allowed to consider
Vector DB → decides what it costs to keep all of it instantly reachable
Retrieval → decides how much of it actually reaches the model, and how well it's chosen
Every one of those is a quality-vs-cost decision, made quietly, usually without realizing it was a decision at all. And every one of them compounds — a mistake made once at ingestion gets paid for again on every query that touches it, for as long as it stays unfixed.
The LLM call at the end was never really the expensive part on its own. It's the stage that inherits every decision made before it, and charges you for all of them at once, every single time someone asks a question.
Less noise, more action. Series closed — for now.