You build a RAG pipeline. The retrieval mostly works. Queries come back with documents that are in the right neighborhood, but not quite the answer. You tweak the prompt. You add more context. You re-chunk the documents for the third time. Nothing moves the needle much.
The bug isn’t in your prompt. It’s in how you’re embedding text in the first place.
Why Symmetric Models Miss the Point #
Most embedding models treat every input the same way. A query and a passage go through the identical pipeline and land in the same vector space, compared with cosine similarity or dot product – the same mechanics behind most similarity search. That works fine when the two things you’re comparing are structurally the same, like two product descriptions or two versions of a sentence.
It falls apart when they’re not. A question like “how do I handle database migrations” is not the same kind of text as a paragraph from a migrations guide. One is short, interrogative, missing context. The other is dense and declarative. Forcing both through a symmetric encoder and expecting the geometry to line up is asking a lot of the model. According to Zilliz’s explanation of asymmetric vs symmetric embedding architectures, asymmetric setups exist precisely because queries and documents aren’t structurally equivalent – they need different encoding paths to land in a space where similarity actually means something.
One developer’s writeup on fixing RAG retrieval with asymmetric embeddings puts it bluntly: standard embedding models are trained for similarity between equivalent texts, and search is not that. Treating a query as if it’s just a short document is the quiet failure mode behind a lot of “almost works” RAG systems.
Our Own Brush With Representation Mismatch #
We hit a version of this ourselves, on the dedup side of our content pipeline rather than retrieval. Our stored post embeddings turned out to be built from the title plus roughly the first 500 characters of the body – effectively an opening vector, not a whole-post vector. When we ran title-based similarity checks against a new draft, it scored a comfortable 0.55 against an existing post it had substantially restated, well under any dedup threshold. The fix was comparing against content embeddings built from the actual post bodies, not titles.
Different failure, same root cause: what you embed determines what you can meaningfully compare it to. Query-vs-document asymmetry and opening-vs-full-body asymmetry are two faces of the same problem – mismatched representations produce similarity scores that lie to you.
Fixing It: task_type and Purpose-Built Models #
Modern APIs are starting to bake this in. Gemini’s embedding models expose a task_type
parameter that tells the model whether it’s encoding a query or a document, and it encodes them differently as a result. If you’re calling an embedding API and passing the same code path for both sides of your retrieval, check whether the model gives you this knob before you assume the model itself is the problem.
Model Choice Isn’t the Whole Story #
Don’t over-rotate on picking the “best” embedding model, either. Independent benchmarks comparing embedding models from OpenAI, Voyage, and Cohere have generally found the performance gap between these models to be smaller than the gap between different chunking strategies – your retrieval logic matters more than which vendor you pick. Asymmetric handling is part of that retrieval logic, not a separate line item.
Get the encoding path right for queries versus documents first. Then argue about which model to buy.