You build a RAG pipeline. You embed your documents, embed the query, and reach for cosine similarity because every tutorial does. It’s baked into every vector database default, every LangChain example, every embedding API’s suggested distance metric. Nobody stops to ask why.
Cosine similarity measures the angle between two vectors, not their length. According to the Wikipedia definition, it’s bounded between -1 and 1, with 1 meaning the vectors point in exactly the same direction. That’s the whole idea. Direction, not magnitude.
For text embeddings, that sounds reasonable. Two sentences about golden retrievers should point the same way in vector space, regardless of how long or short they are. GeeksforGeeks frames it the same way – a metric for similarity “irrespective of size,” which is exactly why people trust it for document comparison and search. We wrote a whole piece on the math behind why cosine and dot product diverge – worth reading if you want the linear algebra spelled out – in our post on how embedding models rank similarity. The short version: cosine throws away magnitude on purpose. That’s the feature. It’s also the trap.
Where the angle lies #
Here’s the thing nobody tells you when they hand you a vector database and a cosine index: throwing away magnitude means throwing away information your embedding model spent a lot of compute encoding. Some models pack confidence, specificity, or frequency signal into vector length. Normalize that away and you’re comparing shape only, not substance.
That’s fine when your embedding space is well-behaved – evenly distributed, isotropic, no weird clustering. It’s a problem when it’s not, and in practice, embedding spaces are rarely as clean as the demo notebook makes them look.
Negation is the tell #
The clearest failure mode shows up with negation. “The product works great” and “the product doesn’t work great” are opposite claims. Semantically, they should sit far apart. In cosine similarity space, they often don’t.
There’s a documented edge case for this – someone building production AI systems ran into what they called negation inversion, where cosine similarity scores a sentence and its negated counterpart as highly similar instead of dissimilar, because most of the words overlap and the embedding model doesn’t weight the negation token heavily enough to flip the vector’s direction (as described in this engineering writeup on taming AI non-determinism).
Think about what that means for a support-ticket classifier, or a semantic search system pulling FAQ answers. A user searches “returns not accepted after 30 days.” Your system pulls back “returns accepted within 30 days” with a high similarity score, because eight of the nine words match and the model’s geometry doesn’t punish the “not” hard enough. The angle between those two vectors is small. The meaning gap is enormous. Cosine similarity doesn’t see the gap. It only sees the angle.
This is the same category of failure we’ve talked about before in The Trap Nobody Notices Until Output Breaks – a metric that looks fine in testing, passes every sanity check, and then quietly returns garbage in production because the failure mode only shows up on inputs your test set didn’t include.
Magnitude carries information you just threw away #
Dot product doesn’t normalize. It multiplies magnitude by magnitude by the cosine of the angle. That means a long, dense, high-confidence embedding scores higher against a query than a short, vague one – even at the same angle.
Whether that’s what you want depends entirely on what your embedding model was trained to do. Some models – the newer sentence-transformer variants trained explicitly for retrieval – are tuned so that dot product on unnormalized vectors is the intended similarity function, not cosine. If you’re running cosine similarity on an embedding model that was trained expecting dot product, you’re not measuring what the model’s authors optimized for. You’re measuring something adjacent to it and hoping it’s close enough.
This is the same pattern we flagged in The Fine-Tuning Trap: the default configuration ships with the framework, everyone assumes it’s the right choice because it’s the default, and the actual mismatch between tool and task only surfaces once you’re debugging weird production behavior at 2am. Cosine similarity is the retrieval-system version of that same trap.
What to check before you ship #
Before you lock in cosine similarity for your search or RAG system, check three things.
First, read your embedding model’s card or paper. Does it say cosine or dot product is the intended metric? Most modern retrieval-tuned models will tell you explicitly. If it says dot product and you’re running cosine, that’s your bug, right there.
Second, build a small negation test set. Take twenty sentences from your actual domain, negate each one, and check the similarity score between the pair. If negated pairs are scoring above 0.7 cosine similarity, your retrieval system will surface contradictory results and you won’t notice until a customer does.
Third, if you’re doing semantic search over short queries and long documents, check whether length is correlating with relevance in your results. If your top results are consistently the longest documents regardless of topic, magnitude information got lost somewhere in normalization and you’re seeing an artifact, not a signal.
None of this requires retraining anything. It requires actually testing the metric you inherited from a tutorial instead of assuming it’s correct because everyone uses it.
The metric is a tool, not a default #
Cosine similarity isn’t wrong. It’s the right choice for a specific class of problem – comparing document direction independent of length, which matters a lot for things like duplicate detection or topic clustering. It’s the wrong default for retrieval systems where magnitude carries meaning, and it’s a documented liability anywhere negation or contradiction shows up in your data.
The trap isn’t the math. The trap is picking a metric because it’s the one every tutorial uses, without checking whether it matches what your embedding model actually encodes. That’s a five-minute check against your model’s documentation and a twenty-sentence test set. Skip it, and you’ll find out the hard way – in production, when a user searches for the opposite of what your system confidently returns.