Query Rewriting Before Retrieval: The Cheap Recall Win Most Skip

A user types "how do I cancel" into your support bot. Your retriever embeds those three words, runs a cosine search, and hands the model five chunks about cancelling a payment, cancelling a meeting invite, and the word "cancel" appearing in a changelog. The chunk about cancelling a subscription, the thing the user actually wanted, ranked seventh. The model answers from what it got. The answer is wrong, and nobody on your team can see why.

The query was the problem. It was too short, too ambiguous, and missing the context the corpus was written with. Most teams reach for a bigger embedding model or a reranker to fix this. Both help. Both are more expensive than the thing that fixes the query before it ever hits the index: query rewriting.

Query rewriting sits in front of retrieval. You take the raw user query, run it through one cheap LLM call, and search with something better. Two patterns carry most of the win: multi-query expansion and step-back rewriting. Here is how each works, what it costs, and how to decide whether the latency is worth it.

Embedding the user's literal words assumes the user phrased the question the way the document phrased the answer. They almost never do.

Users write short, lexically thin queries. "covid policy", "refund", "how do I cancel". Your documents are written by someone else, months earlier, with different vocabulary and full context. The embedding of "how do I cancel" lands in a neighborhood crowded with every cancellation in the corpus. The single relevant chunk is in there, but it is not at the top, and your top-k cut throws it away.

Rewriting fixes the mismatch on the query side, where it is cheap, instead of the index side, where it is not. You are reshaping the search key so it lands closer to the answer.

The idea: one query is one shot at the index. Generate several phrasings of the same intent, search with each, then merge the results. More phrasings means more lexical and semantic surface area, which means a higher chance one of them lands near the right chunk.

You ask the LLM for a handful of variants, run each as its own search, and fuse the rankings with reciprocal rank fusion. RRF rewards documents that several variants agree on, so a chunk that shows up in three of four searches floats to the top even if no single search ranked it first.

from collections import defaultdict
from openai import OpenAI

client = OpenAI()


MULTIQ_PROMPT = """Generate 4 alternative phrasings of the
user's query. Vary the vocabulary and specificity. Each must
stand alone (no pronouns, no "this", no "above").
Return one per line, no numbering."""

def expand(query: str) -> list[str]:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": MULTIQ_PROMPT},
            {"role": "user", "content": query},
        ],
        temperature=0.4,
    )
    content = resp.choices[0].message.content or ""
    lines = content.split("\n")
    variants = [l.strip() for l in lines if l.strip()]
    return [query] + variants  # keep the original

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

def retrieve_multi_query(query: str, k: int = 10) -> list[str]:
    variants = expand(query)
    rankings = [vector_search(v, k=50) for v in variants]
    return rrf(rankings)[:k]

Two details decide whether this helps or just burns tokens. Always keep the original query in the set; the LLM sometimes drifts and the original is your anchor. And use RRF, not a union-and-dedupe. A plain union throws away rank order, which is the only signal that tells you which agreed-upon document to trust.

Multi-query earns its keep on vague and concept-style queries, the ones where the user's intent is fuzzy and a few rephrasings explore it. The variants run in parallel, so the latency cost is one LLM call plus the slowest of N searches, not N searches end to end.

Step-back goes the other direction. Instead of more variants of the same question, you ask the LLM to back up one level of abstraction and pose the broader question first.

The technique comes from a 2023 Google DeepMind paper, Take a Step Back, which found that prompting a model to reason about a general principle before the specific question improved its answers on reasoning benchmarks. The same move helps retrieval. A narrow query like "what is the late-payment penalty for tier 3 enterprise accounts" is so specific that the matching chunk has to contain almost those exact words. The step-back version, "how does billing handle late payments across account tiers", retrieves the policy section that actually defines the penalty in context.

STEP_BACK_PROMPT = """Given a specific user question, write the
broader, more general question whose answer would contain the
context needed to answer the specific one. Return only the
general question, nothing else."""

def step_back(query: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": STEP_BACK_PROMPT},
            {"role": "user", "content": query},
        ],
        temperature=0.0,
    )
    return resp.choices[0].message.content.strip()

def retrieve_step_back(query: str, k: int = 10) -> list[str]:
    broad = step_back(query)
    specific_hits = vector_search(query, k=k)
    broad_hits = vector_search(broad, k=k)
    return rrf([specific_hits, broad_hits])[:k]

Notice the retrieve function searches with both the original and the step-back query and fuses them. You want the broad context chunk and the specific chunk if it exists. The step-back query pulls in the section that frames the answer; the original pulls in the exact match when there is one. Together they give the model the clause plus the context it sits in.

Step-back shines on narrow, jargon-heavy queries against structured documents: contracts, policy manuals, technical specs. The places where the answer needs surrounding context to mean anything.

Published numbers from the source work and from RAG eval surveys give a consistent shape, not a single magic figure. The step-back paper reported double-digit accuracy gains on reasoning tasks. Multi-query expansion with RRF is a long-standing recall improvement in IR literature, with the size of the gain depending heavily on how thin the original queries are.

The honest framing: rewriting moves recall most when your queries are short, vague, or written in different vocabulary than your corpus. It moves recall least when queries are already long, specific, and lexically close to the documents. If your users paste full sentences that mirror your docs, you may see almost nothing. If they type three words into a search box, the gain can be large. You will not know which case you are in until you measure it on your own query log.

That measurement is the part most teams skip, and it is why they argue about rewriters in the abstract. Pull 200 real queries, label the gold documents by hand, and run recall@10 with and without the rewrite. The week of labeling pays for itself the first time it stops you from shipping a rewriter that does nothing for your traffic.

Recall is half the decision. The other half is the LLM call you just added to the hot path.

The rewrite is one extra LLM round trip, typically 250 to 350 ms on a small model like gpt-4o-mini

. The added searches run in parallel, so they cost you the slowest one, not their sum. For an interactive chatbot that already waits on a generation call, 300 ms of rewrite is usually invisible next to the seconds the answer takes to stream. For a high-QPS, low-latency search box, 300 ms is a lot, and you should test whether a reranker on raw candidates closes the same gap for less.

Three ways to keep the cost down:

If you ship one rewriter, ship multi-query expansion with RRF, gated on query length, with a rewrite cache. It almost never makes recall worse, the parallel searches keep latency bounded, and the cache cuts the steady-state cost to near zero. Add step-back when your corpus is structured and your queries are narrow.

Then measure. The rewrite that helps your traffic is an empirical question, not a default someone on Reddit picked for a different corpus. Build the 200-query eval once, run it on every change, and you can swap rewriters without guessing.

The team in the opening still ships the bot that quotes the wrong cancellation. The fix was one LLM call in front of the index they already had.

Query rewriting is the front of a longer pipeline. The RAG Pocket Guide covers it alongside the chunking, hybrid retrieval, and reranking patterns that decide what your rewritten query actually finds — with the eval methodology to tell whether any of it moved recall on your corpus. If you are tuning retrieval and want one mental model instead of a stack of blog posts, that is what it is for.