The Gap Nobody Names

Most RAG pipelines fail because they use symmetric embedding models for asymmetric query-document pairs, according to Zilliz and developer Mario Ottmann. The fix is to use asymmetric embeddings or APIs like Gemini's task_type parameter, and retrieval logic matters more than model choice.

You build a RAG pipeline. The retrieval mostly works. Queries come back with documents that are in the right neighborhood, but not quite the answer. You tweak the prompt. You add more context. You re-chunk the documents for the third time. Nothing moves the needle much. The bug isn’t in your prompt. It’s in how you’re embedding text in the first place. Why Symmetric Models Miss the Point Most embedding models treat every input the same way. A query and a passage go through the identical pipeline and land in the same vector space, compared with cosine similarity or dot product – the same mechanics behind most similarity search. That works fine when the two things you’re comparing are structurally the same, like two product descriptions or two versions of a sentence. It falls apart when they’re not. A question like “how do I handle database migrations” is not the same kind of text as a paragraph from a migrations guide. One is short, interrogative, missing context. The other is dense and declarative. Forcing both through a symmetric encoder and expecting the geometry to line up is asking a lot of the model. According to Zilliz’s explanation of asymmetric vs symmetric embedding architectures https://zilliz.com/ai-faq/what-is-the-difference-between-asymmetric-and-symmetric-embedding-architectures , asymmetric setups exist precisely because queries and documents aren’t structurally equivalent – they need different encoding paths to land in a space where similarity actually means something. One developer’s writeup on fixing RAG retrieval with asymmetric embeddings https://marioottmann.com/articles/asymmetric-embeddings-rag-retrieval puts it bluntly: standard embedding models are trained for similarity between equivalent texts, and search is not that. Treating a query as if it’s just a short document is the quiet failure mode behind a lot of “almost works” RAG systems. Our Own Brush With Representation Mismatch We hit a version of this ourselves, on the dedup side of our content pipeline rather than retrieval. Our stored post embeddings turned out to be built from the title plus roughly the first 500 characters of the body – effectively an opening vector, not a whole-post vector. When we ran title-based similarity checks against a new draft, it scored a comfortable 0.55 against an existing post it had substantially restated, well under any dedup threshold. The fix was comparing against content embeddings built from the actual post bodies, not titles. Different failure, same root cause: what you embed determines what you can meaningfully compare it to. Query-vs-document asymmetry and opening-vs-full-body asymmetry are two faces of the same problem – mismatched representations produce similarity scores that lie to you. Fixing It: task type and Purpose-Built Models Modern APIs are starting to bake this in. Gemini’s embedding models expose a task type parameter that tells the model whether it’s encoding a query or a document, and it encodes them differently as a result. If you’re calling an embedding API and passing the same code path for both sides of your retrieval, check whether the model gives you this knob before you assume the model itself is the problem. Model Choice Isn’t the Whole Story Don’t over-rotate on picking the “best” embedding model, either. Independent benchmarks comparing embedding models from OpenAI, Voyage, and Cohere have generally found the performance gap between these models to be smaller than the gap between different chunking strategies – your retrieval logic matters more than which vendor you pick. Asymmetric handling is part of that retrieval logic, not a separate line item. Get the encoding path right for queries versus documents first. Then argue about which model to buy.