{"slug": "context-compression-before-the-llm-cutting-tokens-without-cutting-recall", "title": "Context Compression Before the LLM: Cutting Tokens Without Cutting Recall", "summary": "A developer describes context compression as a technique to reduce token costs and improve LLM answer quality by filtering retrieved text before generation. Extractive compression keeps verbatim sentences scoring above a threshold, while abstractive compression uses a small model to generate query-focused summaries. The post warns that naive sentence-level filtering can break references and recommends using neighbor windows or cross-encoder rerankers.", "body_md": "You retrieve the top 10 chunks, paste them into the prompt, and send it to the model. Each chunk is 400 tokens. That is 4,000 tokens of context for a question whose answer lives in two sentences buried in chunk 6. You pay for all 4,000 on input. You also pay a quieter tax: the model has to find the answer inside a wall of near-miss text, and longer contexts degrade answer quality even when the right fact is present.\n\nStanford's \"Lost in the Middle\" work showed it clearly. As input context grows, models reliably use information at the start and end and lose track of facts stuck in the middle ([Liu et al., 2023](https://arxiv.org/abs/2307.03172)). So the chunk that ranked sixth, sitting in the middle of your prompt, is exactly where the model is weakest.\n\nContext compression is the layer that sits between retrieval and generation. You retrieve generously, then squeeze the retrieved set down to the part that earns its place in the prompt. Two families do this: extractive and abstractive. They make different trade-offs, and most teams pick the wrong one for their data.\n\nExtractive compression scores each unit of retrieved text against the query and keeps only the units that clear a bar. The text you keep is verbatim from the source. Nothing is rewritten, so nothing is hallucinated into the context.\n\nThe simplest version works at the sentence level. Split each chunk into sentences, embed each sentence and the query, keep the sentences whose similarity beats a threshold.\n\n``` python\nimport numpy as np\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"BAAI/bge-small-en-v1.5\")\n\ndef split_sentences(text: str) -> list[str]:\n    # Cheap splitter; swap for spaCy on messy text.\n    parts = text.replace(\"\\n\", \" \").split(\". \")\n    return [p.strip() for p in parts if p.strip()]\n\ndef compress_extractive(\n    query: str,\n    chunks: list[str],\n    keep: float = 0.5,\n) -> str:\n    sents = []\n    for c in chunks:\n        sents.extend(split_sentences(c))\n    q_vec = model.encode(query, normalize_embeddings=True)\n    s_vecs = model.encode(sents, normalize_embeddings=True)\n    scores = s_vecs @ q_vec\n    n = max(1, int(len(sents) * keep))\n    top = np.argsort(scores)[::-1][:n]\n    # Restore reading order so prose stays coherent.\n    top_sorted = sorted(top.tolist())\n    return \" \".join(sents[i] for i in top_sorted)\n```\n\nThe `keep`\n\nratio is your dial. At `keep=0.5`\n\nyou drop half the sentences and roughly halve token cost on the context. The sentences you keep are the originals, so a citation that points back to the source still lines up word for word.\n\nThe risk with sentence-level filtering is reference breakage. A sentence like \"It expires after 30 days\" scores low against the query \"what is the refund policy\" because it shares no keywords, even though it carries the actual answer. You cut it, and the model loses the qualifier. The fix is to keep a small window around every retained sentence so dangling pronouns and follow-on clauses survive.\n\n``` python\ndef with_neighbors(\n    sents: list[str],\n    kept_idx: list[int],\n    radius: int = 1,\n) -> list[int]:\n    keep = set()\n    for i in kept_idx:\n        for j in range(i - radius, i + radius + 1):\n            if 0 <= j < len(sents):\n                keep.add(j)\n    return sorted(keep)\n```\n\nA trained extractor does better than cosine similarity. Models like `bge-reranker`\n\nscore query-sentence relevance as a cross-encoder, reading both together instead of comparing two separate embeddings. That catches the \"it expires after 30 days\" case more often, because the reranker sees the query and the sentence in one forward pass. It costs more per sentence, so run it on the candidate set after a cheap embedding filter, not on every sentence in the corpus.\n\nAbstractive compression sends the retrieved chunks to a small, fast model and asks it to write a query-focused summary. The output is new text. That is the appeal and the danger.\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI()\n\nCOMPRESS_PROMPT = \"\"\"You are a context compressor. Given a\nquestion and source passages, write a concise summary that\nkeeps every fact relevant to the question. Copy numbers,\nnames, and dates verbatim. Do not add facts. If a passage is\nirrelevant, omit it. If nothing is relevant, return exactly:\nNO_RELEVANT_CONTEXT\"\"\"\n\ndef compress_abstractive(query: str, chunks: list[str]) -> str:\n    joined = \"\\n\\n---\\n\\n\".join(chunks)\n    resp = client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        messages=[\n            {\"role\": \"system\", \"content\": COMPRESS_PROMPT},\n            {\"role\": \"user\",\n             \"content\": f\"Question: {query}\\n\\n{joined}\"},\n        ],\n        temperature=0.0,\n    )\n    return resp.choices[0].message.content\n```\n\nThe compression ratio here can be far higher than extractive. The model can fold five paragraphs that circle the same fact into one sentence. On verbose corpora (support transcripts, meeting notes, legal boilerplate) abstractive can land in the 50-75% range where extractive is bounded nearer 30-50% by how many sentences you keep. Treat both as rough, corpus-dependent rules of thumb, not guaranteed figures.\n\nThe cost is twofold. First, you add an LLM call before the answer call, which adds latency and spend. Second, the summarizer can drop a qualifier or smooth two facts into one wrong fact. \"The discount is 10% for orders over $500\" can come back as \"there is a 10% discount\" once the threshold gets summarized away. The mitigations: temperature 0, an explicit instruction to copy numbers and dates verbatim, and a `NO_RELEVANT_CONTEXT`\n\nescape hatch so the model is allowed to return nothing instead of inventing a connection.\n\nPick on three axes: faithfulness, ratio, and cost.\n\n| Extractive | Abstractive | |\n|---|---|---|\n| Faithful to source | high (verbatim) | medium (rewritten) |\n| Typical token cut | 30-50% | 50-75% |\n| Hallucination risk | none | real |\n| Added latency | low (no LLM) | one extra LLM call |\n| Citation alignment | exact | approximate |\n\nIf your domain punishes wrong facts (legal, medical, finance, anything with a number that matters), start extractive. The verbatim guarantee is the point. If your corpus is verbose and repetitive and the cost of a near-miss summary is low, abstractive buys more headroom.\n\nThere is a hybrid that gets most of both. Run extractive first to throw away the obviously irrelevant sentences, then run abstractive on what survives. The summarizer sees a cleaner, shorter input, so it hallucinates less and costs less, and you still get the high compression ratio.\n\nA compressor that cuts 60% of tokens and 5% of correct answers is a bad trade you will not notice until users complain. The number that matters is whether the compressed context still contains the facts the model needs. That is context recall, and you can measure it directly.\n\nTake your labeled eval set: question, retrieved chunks, and the gold answer. Run compression. Then check whether the gold facts survived.\n\n``` php\ndef context_recall(\n    compressed: str,\n    gold_facts: list[str],\n) -> float:\n    if not gold_facts:\n        return 1.0\n    hits = sum(\n        1 for f in gold_facts\n        if f.lower() in compressed.lower()\n    )\n    return hits / len(gold_facts)\n```\n\nExact substring matching is crude; for paraphrase-tolerant scoring, an LLM judge or an entailment model reads \"does this context support this fact\" per gold fact. Either way, the loop is the same: sweep the `keep`\n\nratio (or compare extractive vs abstractive) and plot context recall against token cost. You are looking for the knee in the curve, the point where dropping more tokens starts costing real answers.\n\nRun it against the same eval you use for retrieval, not a fresh cherry-picked set. The whole reason compression is worth doing is that it sits on the critical path between your retriever and your bill. Treat it like the rest of the pipeline: instrument it, sweep the dial, keep the setting that wins on your data instead of the one a blog post liked.\n\nThe order is fixed. Retrieve more than you need, rerank, compress, generate.\n\nRetrieve wide (top 20 instead of top 5) because compression lets you afford it. Rerank so the best chunks are densest at the top, which makes both compressors more effective. Compress to cut the prompt down to the part that earns its tokens. Then generate, with a shorter, sharper context that the model reads more reliably because the answer is no longer buried in the middle.\n\nSkip compression when your retrieved context is already small and on-target, or when you have a long-context model and token cost is not your constraint. The layer earns its place when you are retrieving generously, paying per input token, and watching answer quality slip as context grows. That is most production RAG.\n\nThe [RAG Pocket Guide](https://www.amazon.com/dp/B0GX2YDC5Z) covers this layer alongside retrieval, chunking, and reranking, with the eval methodology to pick a compression strategy on your own corpus instead of guessing. If your prompts are fat and your answers are slipping, compression is the cheapest lever you are probably not pulling yet.", "url": "https://wpnews.pro/news/context-compression-before-the-llm-cutting-tokens-without-cutting-recall", "canonical_source": "https://dev.to/gabrielanhaia/context-compression-before-the-llm-cutting-tokens-without-cutting-recall-9hh", "published_at": "2026-06-13 22:21:50+00:00", "updated_at": "2026-06-13 22:50:43.242389+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research", "developer-tools"], "entities": ["Stanford", "BAAI", "bge-small-en-v1.5", "OpenAI", "Liu et al.", "bge-reranker"], "alternates": {"html": "https://wpnews.pro/news/context-compression-before-the-llm-cutting-tokens-without-cutting-recall", "markdown": "https://wpnews.pro/news/context-compression-before-the-llm-cutting-tokens-without-cutting-recall.md", "text": "https://wpnews.pro/news/context-compression-before-the-llm-cutting-tokens-without-cutting-recall.txt", "jsonld": "https://wpnews.pro/news/context-compression-before-the-llm-cutting-tokens-without-cutting-recall.jsonld"}}