RAG Context Budget: Pack Evidence Without Burying the Answer

A developer outlines a vendor-neutral "RAG context budget" policy for retrieval-augmented generation that caps prompt space for instructions, conversation, evidence, answer reserve, and a safety margin, rather than simply raising top_k. The approach retrieves a broad candidate set, reranks it, and admits only diverse, non-overlapping evidence that earns its token cost, with deterministic query-class routing (point lookup, how-to, comparison, aggregation, troubleshooting) deciding packing rules. The author argues that larger context windows are capacity, not a retrieval strategy, and that over-stuffing prompts buries the useful passage.

When a RAG answer is wrong, the reflex is often to raise top k . More retrieved chunks feels safer. In production, it can do the opposite: the one useful passage gets surrounded by repeated, stale, or merely related text, while the model has less room to answer. A large context window is capacity, not a retrieval strategy. Treat the input as a RAG context budget : reserve room for the answer, retrieve widely, then admit only the evidence that earns its token cost. This guide shows a vendor-neutral policy you can implement around any vector store and model API. Retrieval and generation have different jobs. A retriever should find a broad candidate set so a good passage is not missed. The prompt assembler should build a small, diverse, attributable evidence set that a model can use. Those are not the same top k . For example, a support assistant may retrieve 30 candidates to protect recall, rerank the best 12, and send four short, non-overlapping sections to the model. A policy question may need the newest authoritative policy plus one exception. A “list every affected region” question may need more coverage and a different answer shape. Sending every candidate creates four problems: Recent retrieval systems increasingly select useful highlights rather than whole pages. That direction is sensible, but it does not remove the engineering decision: your application still needs an explicit policy for what may enter the generation prompt. Start from the smallest context you want to support reliably, not from a model’s maximum window. Reserve fixed and variable space first: | Allocation | What it contains | Why it needs a cap | |---|---|---| | Instructions | safety rules, format, tool policy | grows with every feature request | | Conversation | relevant turns and user constraints | can become the dominant input | | Evidence | retrieved passages and metadata | must compete fairly with the answer | | Answer reserve | response plus citations | prevents truncation and rushed output | | Safety margin | tokenizer and message-format overhead | avoids boundary failures | The basic equation is: evidence budget = input limit - instruction tokens - conversation tokens - answer reserve - safety margin Use the provider tokenizer for the final count. A rough estimator is fine while choosing candidates, but do not assume words equal tokens or that chat-message formatting is free. The budget is a product decision too. If the user asks for a comparison with sources, reserve enough output for distinctions and citations. If the task returns a short JSON classification, the answer reserve can be smaller. A single global budget hides that trade-off. A practical context packer has four gates: This means a lower-scoring chunk can win a place because it covers a missing part of a multi-part question. Likewise, a high-scoring duplicate should lose once its fact is already represented. One cap cannot serve every intent. Begin with a small classifier that chooses a policy, and log its decision: | Query class | Primary goal | Packing rule | |---|---|---| | Point lookup | precise fact | one authoritative section; add a second only for verification | | How-to | ordered steps | preserve a complete procedure and its prerequisites | | Comparison | distinctions | reserve space for each option and a shared criterion | | Aggregation | coverage | collect distinct sources; state what may be incomplete | | Troubleshooting | diagnosis | favor version, error, and environment matches over generic advice | Start with deterministic routing based on obvious cues and test data. You can later use a classifier, but its output should remain observable and easy to override. A wrong class should fail safely by returning a narrower answer, asking a question, or disclosing limits—not by stuffing every result into the prompt. The example below accepts already-authorized, reranked candidates. It leaves a fixed answer reserve, rejects near-duplicates, and gives each source a small diversity allowance. It is intentionally simple: production systems also need timeout handling, tracing, and model-specific token counting. python from dataclasses import dataclass @dataclass class Chunk: id: str source id: str tokens: int relevance: float novelty: float 0..1; lower means it repeats accepted evidence authority: float 0..1; derived from your source policy text: str def pack context candidates, evidence budget, per source limit=2 : accepted, used, source counts = , 0, {} Relevance gets us close; novelty and authority stop a score-only pack. ranked = sorted candidates, key=lambda c: 0.60 c.relevance + 0.25 c.novelty + 0.15 c.authority, reverse=True, for chunk in ranked: if chunk.novelty < 0.20: continue if source counts.get chunk.source id, 0 = per source limit: continue if used + chunk.tokens evidence budget: continue accepted.append chunk used += chunk.tokens source counts chunk.source id = source counts.get chunk.source id, 0 + 1 return accepted, used Do not mistake this heuristic for ground truth. The purpose is to make an admission policy visible and testable. Your novelty signal could be cosine-distance from accepted chunks, heading overlap, a deduplication hash, or an evaluator. Your authority signal should come from a source registry rather than an LLM’s opinion. If one excellent section is larger than the budget, do not blindly slice it at a character boundary. Prefer a precomputed passage, a parent section with an anchored excerpt, or a targeted extraction step. Keep the citation identifier with every excerpt so the answer can point back to what it used. The most subtle failure is calculating evidence first and letting the answer take whatever remains. That makes answer quality depend on retrieval volume. Instead, choose an answer contract first. A short factual answer may reserve 300 tokens. A troubleshooting response with diagnosis, safe steps, and cited evidence may reserve 900 or more. A structured output contract should reserve room for its full schema and error case. Set an evidence ceiling below the model limit even if you could fit more. This makes cost and latency predictable, and it makes behavior portable when you later route a request to a smaller model. If the candidate pack does not fit, apply a known fallback: Avoid silently dropping the last chunks. Tail-drop creates position bias and makes results depend on incidental sort order. Context packing is also a safety boundary. A retrieved document is evidence, not an instruction. Label it as untrusted source material, keep it outside the system instruction channel, and do not let its text decide which tools are available. This matters even for an internal knowledge base: imported tickets, copied documentation, and web pages can contain text that looks like a command. Use a small envelope for every admitted passage, such as a source ID, title, version or fetch time, access label, and the excerpt itself. The model can cite the ID, while application code resolves it to a safe link. Do not invite the model to invent an origin from raw text. Tool results deserve their own cap. A database query that returns 5,000 rows is not a better answer than a bounded aggregation with a continuation path. For tools, define a response schema, a maximum row or byte count, a pagination contract, and a redaction step before results reach the packer. If an action needs additional evidence, make a second bounded call instead of carrying every prior result through the conversation. This separation makes incident review practical. You can answer three useful questions from a trace: which instruction governed the response, which sources supported it, and which tool result changed its state. A single concatenated prompt cannot answer those questions reliably. An answer score alone cannot tell you whether the failure came from retrieval, reranking, packing, or generation. Add a trace for every sampled request: { "query class": "troubleshooting", "evidence budget": 2600, "candidate count": 30, "reranked count": 12, "accepted chunk ids": "docs-18 4", "release-7 2" , "evidence tokens": 1412, "answer reserve": 900, "dropped": {"duplicate": 5, "budget": 4, "source cap": 1} } Then evaluate representative questions by query class. For each, record whether the expected evidence was retrieved, admitted, cited, and actually supports the answer. This separates four common diagnoses: | Observation | Likely issue | Next action | |---|---|---| | Needed source never appears | retrieval recall | improve indexing, filters, or candidate depth | | Needed source is retrieved but excluded | packing/reranking | inspect budget, diversity, and score rules | | Source is included but ignored | prompt or ordering | improve instructions, placement, or answer format | | Source is cited but answer is wrong | generation/verification | add claim checks or require abstention | Measure cost, latency, groundedness, and task success together. Lowering context tokens is not a win if correct answers disappear. Conversely, sending more text is not a win if it only increases cost and makes attribution worse. Use a shadow trace first: calculate the proposed pack while still sending the current one. Compare token counts and whether expected evidence would have been admitted. Next, run a small traffic cohort with a kill switch. Test deliberately awkward cases: duplicated policy pages, an old and a new version, a query that needs two documents, a broad “list all” request, and a long conversation with a short current question. Include authorization changes in this suite. A perfect pack that leaks a cross-tenant passage is still a failure. The goal is not the smallest possible prompt. It is the smallest sufficient evidence set for a task, with a predictable answer reserve and a record of why each passage was admitted. It is the portion of an LLM request deliberately reserved for retrieved evidence after instructions, relevant conversation, the answer, and a safety margin have been allocated. It turns context assembly into an explicit policy instead of a fixed top k setting. There is no universal number. Retrieve enough candidates for recall, then evaluate how many diverse passages each query class needs. A point lookup may need one authoritative section; an aggregation question may need several. Use a token ceiling and test against real tasks. Not automatically. First find out whether the needed source was missing, retrieved but excluded, included but ignored, or cited incorrectly. Raising top k helps only the first case and can make the other three worse. Store stable source and section identities, detect overlap or semantic similarity among candidates, and cap passages per source. Preserve the strongest chunk and its provenance rather than sending several near-identical versions of the same fact. No. Larger windows can hold more material, but they do not make every passage relevant, authorized, current, or easy for a model to use. Packing still controls cost, attention, provenance, and answer space. Log the query class, token allocations, candidate and accepted IDs, source versions, drop reasons, latency, and the final citations. Avoid storing sensitive raw text unless retention and access rules explicitly allow it.