Context Compression Before the LLM: Cutting Tokens Without Cutting Recall

You retrieve the top 10 chunks, paste them into the prompt, and send it to the model. Each chunk is 400 tokens. That is 4,000 tokens of context for a question whose answer lives in two sentences buried in chunk 6. You pay for all 4,000 on input. You also pay a quieter tax: the model has to find the answer inside a wall of near-miss text, and longer contexts degrade answer quality even when the right fact is present.

Stanford's "Lost in the Middle" work showed it clearly. As input context grows, models reliably use information at the start and end and lose track of facts stuck in the middle (Liu et al., 2023). So the chunk that ranked sixth, sitting in the middle of your prompt, is exactly where the model is weakest.

Context compression is the layer that sits between retrieval and generation. You retrieve generously, then squeeze the retrieved set down to the part that earns its place in the prompt. Two families do this: extractive and abstractive. They make different trade-offs, and most teams pick the wrong one for their data.

Extractive compression scores each unit of retrieved text against the query and keeps only the units that clear a bar. The text you keep is verbatim from the source. Nothing is rewritten, so nothing is hallucinated into the context.

The simplest version works at the sentence level. Split each chunk into sentences, embed each sentence and the query, keep the sentences whose similarity beats a threshold.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

def split_sentences(text: str) -> list[str]:
    parts = text.replace("\n", " ").split(". ")
    return [p.strip() for p in parts if p.strip()]

def compress_extractive(
    query: str,
    chunks: list[str],
    keep: float = 0.5,
) -> str:
    sents = []
    for c in chunks:
        sents.extend(split_sentences(c))
    q_vec = model.encode(query, normalize_embeddings=True)
    s_vecs = model.encode(sents, normalize_embeddings=True)
    scores = s_vecs @ q_vec
    n = max(1, int(len(sents) * keep))
    top = np.argsort(scores)[::-1][:n]
    top_sorted = sorted(top.tolist())
    return " ".join(sents[i] for i in top_sorted)

The keep

ratio is your dial. At keep=0.5

you drop half the sentences and roughly halve token cost on the context. The sentences you keep are the originals, so a citation that points back to the source still lines up word for word.

The risk with sentence-level filtering is reference breakage. A sentence like "It expires after 30 days" scores low against the query "what is the refund policy" because it shares no keywords, even though it carries the actual answer. You cut it, and the model loses the qualifier. The fix is to keep a small window around every retained sentence so dangling pronouns and follow-on clauses survive.

def with_neighbors(
    sents: list[str],
    kept_idx: list[int],
    radius: int = 1,
) -> list[int]:
    keep = set()
    for i in kept_idx:
        for j in range(i - radius, i + radius + 1):
            if 0 <= j < len(sents):
                keep.add(j)
    return sorted(keep)

A trained extractor does better than cosine similarity. Models like bge-reranker

score query-sentence relevance as a cross-encoder, reading both together instead of comparing two separate embeddings. That catches the "it expires after 30 days" case more often, because the reranker sees the query and the sentence in one forward pass. It costs more per sentence, so run it on the candidate set after a cheap embedding filter, not on every sentence in the corpus.

Abstractive compression sends the retrieved chunks to a small, fast model and asks it to write a query-focused summary. The output is new text. That is the appeal and the danger.

from openai import OpenAI

client = OpenAI()

COMPRESS_PROMPT = """You are a context compressor. Given a
question and source passages, write a concise summary that
keeps every fact relevant to the question. Copy numbers,
names, and dates verbatim. Do not add facts. If a passage is
irrelevant, omit it. If nothing is relevant, return exactly:
NO_RELEVANT_CONTEXT"""

def compress_abstractive(query: str, chunks: list[str]) -> str:
    joined = "\n\n---\n\n".join(chunks)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": COMPRESS_PROMPT},
            {"role": "user",
             "content": f"Question: {query}\n\n{joined}"},
        ],
        temperature=0.0,
    )
    return resp.choices[0].message.content

The compression ratio here can be far higher than extractive. The model can fold five paragraphs that circle the same fact into one sentence. On verbose corpora (support transcripts, meeting notes, legal boilerplate) abstractive can land in the 50-75% range where extractive is bounded nearer 30-50% by how many sentences you keep. Treat both as rough, corpus-dependent rules of thumb, not guaranteed figures.

The cost is twofold. First, you add an LLM call before the answer call, which adds latency and spend. Second, the summarizer can drop a qualifier or smooth two facts into one wrong fact. "The discount is 10% for orders over $500" can come back as "there is a 10% discount" once the threshold gets summarized away. The mitigations: temperature 0, an explicit instruction to copy numbers and dates verbatim, and a NO_RELEVANT_CONTEXT

escape hatch so the model is allowed to return nothing instead of inventing a connection.

Pick on three axes: faithfulness, ratio, and cost.

If your domain punishes wrong facts (legal, medical, finance, anything with a number that matters), start extractive. The verbatim guarantee is the point. If your corpus is verbose and repetitive and the cost of a near-miss summary is low, abstractive buys more headroom.

There is a hybrid that gets most of both. Run extractive first to throw away the obviously irrelevant sentences, then run abstractive on what survives. The summarizer sees a cleaner, shorter input, so it hallucinates less and costs less, and you still get the high compression ratio.

A compressor that cuts 60% of tokens and 5% of correct answers is a bad trade you will not notice until users complain. The number that matters is whether the compressed context still contains the facts the model needs. That is context recall, and you can measure it directly.

Take your labeled eval set: question, retrieved chunks, and the gold answer. Run compression. Then check whether the gold facts survived.

def context_recall(
    compressed: str,
    gold_facts: list[str],
) -> float:
    if not gold_facts:
        return 1.0
    hits = sum(
        1 for f in gold_facts
        if f.lower() in compressed.lower()
    )
    return hits / len(gold_facts)

Exact substring matching is crude; for paraphrase-tolerant scoring, an LLM judge or an entailment model reads "does this context support this fact" per gold fact. Either way, the loop is the same: sweep the keep

ratio (or compare extractive vs abstractive) and plot context recall against token cost. You are looking for the knee in the curve, the point where dropping more tokens starts costing real answers.

Run it against the same eval you use for retrieval, not a fresh cherry-picked set. The whole reason compression is worth doing is that it sits on the critical path between your retriever and your bill. Treat it like the rest of the pipeline: instrument it, sweep the dial, keep the setting that wins on your data instead of the one a blog post liked.

The order is fixed. Retrieve more than you need, rerank, compress, generate.

Retrieve wide (top 20 instead of top 5) because compression lets you afford it. Rerank so the best chunks are densest at the top, which makes both compressors more effective. Compress to cut the prompt down to the part that earns its tokens. Then generate, with a shorter, sharper context that the model reads more reliably because the answer is no longer buried in the middle.

Skip compression when your retrieved context is already small and on-target, or when you have a long-context model and token cost is not your constraint. The layer earns its place when you are retrieving generously, paying per input token, and watching answer quality slip as context grows. That is most production RAG.

The RAG Pocket Guide covers this layer alongside retrieval, chunking, and reranking, with the eval methodology to pick a compression strategy on your own corpus instead of guessing. If your prompts are fat and your answers are slipping, compression is the cheapest lever you are probably not pulling yet.