{"slug": "why-king-man-woman-isn-t-queen", "title": "Why king – man + woman isn't queen", "summary": "A vector-search explainer measured Qwen3-Embedding-0.6B, a small open-weights embedding model run locally, and found that the classic king − man + woman analogy returns \"woman\" at 0.823 cosine similarity ahead of \"queen\" at 0.787 unless the input words are excluded from candidates, as word2vec's analogy evaluation and gensim's most_similar do. The same tests found unrelated sentences scoring 0.12 to 0.37, \"The refund was not approved\" scoring 0.89 against \"The refund was approved\", BM25 reaching 0.93 nDCG@10 on identifier queries versus 0.21 to 0.30 for embedding models, and vec2text reconstructing 92% of 32-token inputs exactly from their embeddings. The author argues cosine similarity is not a percentage and thresholds must be calibrated per model, and that embeddings are lossy summaries requiring the same access controls as the source text.", "body_md": "# Vector Search 101: What an Embedding Actually Is\n\nWhy king - man + woman isn't queen, and seven other things we measured.\n\nAn **embedding** is a fixed-length list of numbers that a model produces for a piece of content, so that content with similar meaning gets vectors pointing in similar directions. Vector search embeds your query with the same model and returns the stored items whose vectors are closest - usually by cosine similarity, the cosine of the angle between two vectors.\n\nThat paragraph is accurate, and it's roughly where most introductions stop. The trouble is that it leaves you with intuitions that are *mostly* right, and the \"mostly\" is where search systems go wrong in production.\n\nSo this post takes a different route to understanding embeddings: it starts from the definition above and then goes through eight surprises that show up as soon as you actually measure. Each one corrects a common intuition, and together they're a much better mental model of what an embedding is than any definition. Let's go over them:\n\nAll numbers below come from [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), a small open-weights model run locally on a laptop, unless stated otherwise. Other modern models differ in the exact numbers, not in the shape of the results.\n\n## Key takeaways\n\n- **An embedding is a lossy summary of what a text is about** : a fixed-length vector - 1,024 numbers for Qwen3-Embedding-0.6B - in which topic dominates and exact details fade.\n- **Cosine similarity isn't a percentage.** Unrelated sentences scored 0.12 to 0.37, so a cutoff like 0.8 means different things on different models; calibrate thresholds per model.\n- **Negation and identifiers barely register.** \"The refund was not approved\" scored 0.89 against \"The refund was approved\", and on identifier queries BM25 reached 0.93 nDCG@10 against 0.21 to 0.30 for embedding models.\n- **Long text blurs, and truncation is silent.** Every unrelated sentence around an answer lowers its score, and past a model's input limit the rest isn't embedded at all - with no error.\n- **A question matches a question better than its answer** , so use the model's query instruction or input type.\n- **Vectors aren't anonymized.** vec2text rebuilt 92% of 32-token inputs exactly from their embeddings, so vectors need the same access controls as the text.\n\n## In this post11 sections\n\n## A 60-second refresher\n\nText goes in, a tokenizer cuts it into sub-word pieces, a transformer turns the pieces into context-aware vectors, and a pooling step collapses those into one vector - 1,024 numbers for this model - normalized to length 1:\n\n```\nvec = model.encode(\"The warranty covers the frame for five years.\")\nvec.shape          # (1024,)\nfloat(vec @ vec)   # 1.0 - unit length, so a dot product *is* cosine similarity\n```\n\nThe model learned to produce these through contrastive training: pairs of texts that belong together (a question and its answer, a title and its article) are pulled together, everything else in the batch is pushed apart. Keep that training objective in mind - almost every surprise below follows from it.\n\n## 1. king - man + woman = ... woman\n\nThe most famous fact about embeddings is that you can do arithmetic with meaning: take the vector for \"king\", subtract \"man\", add \"woman\", and you land on \"queen\". It's in nearly every introduction.\n\nLet's try it, on a 40-word vocabulary that includes royalty, family words, capitals and some distractors:\n\n```\ntarget = E[\"king\"] - E[\"man\"] + E[\"woman\"]\nnearest(target)\n```\n\n| Rank | Word | Cosine | \n|---|---|---|\n| 1 | woman | 0.823 | \n| 2 | king | 0.804 | \n| 3 | queen | 0.787 | \n| 4 | wife | 0.786 | \n\n\"Queen\" is third. The nearest word to king - man + woman is *woman*, followed by *king*.\n\nThe trick in the original demonstrations is that the input words are **excluded** from the candidates - word2vec's analogy evaluation and gensim's `most_similar` both do this. Exclude them and the answer is queen, at 0.787, just ahead of wife at 0.786. Nissim, van Noord and van der Goot wrote a whole paper on this in 2020 (\"Fair is Better than Sensational\"), mostly because some of the more alarming headlines about biased analogies relied on the same exclusion.\n\nThe capital-city version behaves better - paris - france + germany gives berlin at 0.907 with the inputs included - so the effect is real. But the lesson is the right mental model for embeddings: **the arithmetic moves you a little in a direction; it doesn't teleport you to a concept.** Most of what's in a vector is \"what this is about\", and subtracting man doesn't remove much of it.\n\n## 2. Unrelated texts score 0.25, not 0\n\nTake ten sentences with nothing to do with each other - cats, quarterly revenue, a train timetable, photosynthesis, a VPN reset, a cake recipe, Everest, an API, a cellist, a bike warranty - and compute all 45 pairwise similarities:\n\n| Statistic | Cosine | \n|---|---|\n| Lowest | 0.12 | \n| Mean | 0.25 | \n| Highest | 0.37 | \n\nIf vectors were spread uniformly in 1,024 dimensions, unrelated pairs would sit at 0 with a standard deviation of about 1/√1024 ≈ 0.03 - with random vectors, 99% of pairs land within ±0.08. Real models don't use the whole space. Their vectors live in a narrow cone, so *everything* is somewhat similar to everything else. Researchers call this **anisotropy**, and how narrow the cone is varies by model: developers using OpenAI's older ada-002 reported unrelated texts scoring above 0.7.\n\nIt's not an artifact of those ten sentences either: across SciFact's 5,183 scientific abstracts, two random abstracts average 0.26, while a claim and an abstract judged relevant to it average 0.65 - and 95% of relevant pairs score below 0.80.\n\nThe practical consequence: **a cosine similarity is not a percentage.** 0.25 doesn't mean \"25% similar\", and a threshold like \"only return results above 0.8\" means completely different things on different models. Scores are only meaningful relative to other scores from the same model.\n\n## 3. Opposites are neighbors\n\n| Sentence A | Sentence B | Cosine | \n|---|---|---|\n| The refund was approved. | The refund was approved yesterday. | 0.94 | \n| The refund was approved. | The refund was **not** approved. | 0.89 | \n| The refund was approved. | The refund was rejected. | 0.87 | \n| The cat sat on the mat. | A kitten dozed on the rug. | 0.69 | \n\n\"Approved\" and \"not approved\" are closer than two sentences about cats lying on things. This one follows straight from the training objective: training pairs are about *relevance*, and \"the refund was not approved\" is extremely relevant to a question about whether the refund was approved. It's the right document to retrieve - it's just a terrible document to *trust* if you only check that it matched.\n\nEmbeddings are topic detectors first. Negation, polarity and quantity (\"approved\" vs \"rejected\", \"5\" vs \"50\") are faint signals on top.\n\n## 4. Identifiers don't survive\n\nHere's how the tokenizer sees an invoice number:\n\n```\n>>> tok.tokenize(\"INV-20931\")\n['INV', '-', '2', '0', '9', '3', '1']\n```\n\nFive separate digit tokens, summarized into the same 1,024 numbers as everything else. Take five invoices from an invented bike shop and search for \"Was invoice INV-20931 paid?\". The top result was INV-**20913** - the same digits, reordered, on an invoice that was *unpaid*.\n\nTo check this wasn't a fluke, here's a public benchmark - SciFact, 5,183 scientific abstracts - with 200 extra queries that are just an identifier found in one document (things like `bcl-214`), in nDCG@10 where 1.0 means the right document always comes first:\n\n| Search | Normal questions | Identifier queries | \n|---|---|---|\n| BM25 keywords | 0.69 | 0.93 | \n| Qwen3-Embedding-0.6B | 0.70 | 0.23 | \n| Qwen3-Embedding-8B | 0.79 | 0.30 | \n| Cohere Embed Multilingual v3 | 0.70 | 0.21 | \n\nA 13x larger model gets a lot better at questions and barely better at identifiers. That's the signature of a structural limitation rather than a quality gap, and it lines up with a 2025 Google DeepMind paper (Weller et al., \"On the Theoretical Limitations of Embedding-Based Retrieval\") that proves a single vector of a given dimension can only represent so many distinct top-k result sets - and shows strong 7-8B embedding models scoring under 20% recall@100 on a dataset built to hit that limit, where BM25 gets 93.6%.\n\n## 5. Longer text gets blurrier, and the end may not count at all\n\nAn embedding of a passage is a summary of the whole passage. So what happens when the one sentence you care about is surrounded by others? Take \"The warranty covers the frame for five years.\", pad it with unrelated sentences from the same imaginary shop (opening hours, parking, gift cards), and measure its similarity to \"How long is the frame warranty?\":\n\n| Filler sentences added | Tokens | Cosine to the question | \n|---|---|---|\n| 0 | 10 | 0.78 | \n| 1 | 18 | 0.69 | \n| 6 | 54 | 0.66 | \n| 12 | 94 | 0.62 | \n| 48 | 346 | 0.59 | \n\nTwelve filler sentences on their own, with no warranty sentence, score 0.54. With 48 filler sentences around it, the passage that contains the exact answer is only 0.05 above that - barely distinguishable from a passage with nothing relevant in it. This is why **[chunking](https://deployed-ai.io/blog/what-is-chunking/)** - splitting documents into passages before embedding them - matters so much.\n\nAnd there's a sharper version. Every model has a maximum input length, and most libraries truncate silently. Here's a 1,021-token terms-and-conditions document with the warranty clause at the very end:\n\n| Model input limit | Cosine to \"How long is the frame warranty?\" | \n|---|---|\n| 8,192 tokens | 0.57 | \n| 1,024 tokens | 0.57 | \n| 512 tokens | 0.28 | \n| 256 tokens | 0.29 | \n\nAt 512 tokens the clause is cut off, and the document scores *below* the boilerplate on its own (0.32). No error, no warning. Plenty of popular models stop at 512 tokens, and some well-known sentence-transformers models at 256.\n\n## 6. A question matches a question better than its answer\n\nSearch for \"how long is the frame warranty\" against an answer and a near-copy of the question:\n\n| Document | No query instruction | With query instruction | \n|---|---|---|\n| How long is the frame warranty? | 0.98 | 0.87 | \n| The warranty covers the frame for five years. | 0.84 | 0.81 | \n| Frames are made of aluminium. | 0.54 | 0.64 | \n\nSimilarity is symmetric, and the text most similar to a question is... a similar question. That's why many models want you to mark queries and documents differently - Qwen3-Embedding takes an instruction prefix on queries, E5 models want `query:` and `passage:`, Cohere and Voyage take an input type. The instruction narrows the gap (0.87 vs 0.81 instead of 0.98 vs 0.84) but, here, doesn't close it.\n\nIn practice this means FAQ pages, forum threads and ticket titles that *repeat* a question tend to outrank the document that *answers* it. Forget the prefix and it gets worse.\n\n## 7. Same question, other language, can lose to a different question\n\nModern embedding models are multilingual, which is genuinely magical:\n\n| Text | Cosine to \"Where is the train station?\" | \n|---|---|\n| ¿Dónde está la estación de tren? | 0.84 | \n| Where is the bus stop? | 0.75 | \n| Où est la gare ? | 0.72 | \n| איפה תחנת הרכבת? | 0.58 | \n| I would like a coffee, please. | 0.40 | \n\nSpanish, French and Hebrew translations all clearly beat the unrelated sentence. But \"Where is the bus stop?\" - a *different question* in English - beats the exact French translation. Embedding spaces are multilingual, but language is still one of the directions in them, and it's not a small one. For a mixed-language corpus that means same-language results get a head start.\n\n## 8. An embedding isn't an anonymized copy of the text\n\nIt's tempting to think of a vector as a one-way hash of the text - 1,024 opaque floats, surely you can't get the words back. You can, to a large extent. Morris et al.'s vec2text (2023) reconstructs 92% of 32-token inputs *exactly* from their embeddings (for the encoder it was trained to invert), and recovered names from clinical notes. If the text is sensitive, the vectors are sensitive too: they need the same access controls, the same retention and the same deletion.\n\n## Conclusions\n\nPutting the eight together gives a better definition than the one at the top:\n\nAn embedding is a lossy summary of what a piece of text is *about*, in a space where everything is somewhat similar, topic dominates, and exact details - identifiers, negation, the end of long documents - fade or disappear.\n\nThat's not a knock on embeddings. They're the reason a search for \"sales rose by twelve percent\" can find \"revenue grew 12%\" (cosine 0.84, with \"grew 12%\" and \"rose by twelve percent\" sharing no word). It's an argument for using them for what they're good at, and for measuring the rest.\n\nWe ran into most of these while building search for [Deployed](https://deployed-ai.io), and they shaped how it works.\n\nIf you take one thing from this post: before you trust a vector search, embed twenty of your own queries and look at the raw numbers - hopefully with fewer surprises. Thanks for reading and hope you enjoyed!\n\n## Frequently asked questions\n\n## ### What is an embedding?\n\nAn embedding is a fixed-length list of numbers that a model produces for a piece of content, so that content with similar meaning gets vectors pointing in similar directions. A tokenizer cuts the text into sub-word pieces, a transformer turns them into context-aware vectors, and a pooling step collapses those into one vector - 1,024 numbers for Qwen3-Embedding-0.6B - normalized to length 1.\n\n## ### How does vector search work?\n\nVector search embeds the query with the same model that embedded the documents and returns the stored items whose vectors are closest, usually by cosine similarity - the cosine of the angle between two vectors. Embedding models are trained to pull related pairs together, such as a question and its answer, so vector search finds text that shares meaning but not words: \"sales rose by twelve percent\" matches \"revenue grew 12%\" at 0.84.\n\n## ### What is a good cosine similarity threshold?\n\nThere isn't a universal one. With Qwen3-Embedding-0.6B, unrelated sentences scored 0.12 to 0.37, and developers using OpenAI's older ada-002 reported unrelated texts above 0.7. To set a threshold, embed a few hundred real queries, compare the similarity of the right answer with that of the best wrong answer, pick a cut from that distribution - and redo it when you change models.\n\n## ### Does king - man + woman equal queen?\n\nOnly if you leave out the input words. On a 40-word vocabulary, the nearest word to king - man + woman was \"woman\" (0.823), then \"king\" (0.804), then \"queen\" (0.787). word2vec's analogy evaluation and gensim's `most_similar` exclude the input words, which is how the famous demonstration lands on queen. The arithmetic moves a vector a little in a direction; it doesn't jump to a concept.\n\n## ### Why can't vector search find invoice numbers or other IDs?\n\nA tokenizer splits an identifier like INV-20931 into seven tokens, and the embedding summarizes them into the same 1,024 numbers as everything else, so nearly identical IDs look alike: a search for INV-20931 returned INV-20913 first. On identifier queries over SciFact, BM25 keyword search scored 0.93 nDCG@10, against 0.21 to 0.30 for three embedding models - which is why search systems pair the two.\n\n## ### Why does a long passage get a lower similarity score?\n\nAn embedding summarizes the whole passage, so every sentence that isn't the answer dilutes it: a warranty sentence scored 0.78 against its question on its own and 0.59 with 48 unrelated sentences around it, barely above the 0.54 that twelve filler sentences score with no answer at all. Past a model's input limit - 512 tokens for many popular models - most libraries truncate silently, so the end of a document may not count at all. That's why [chunking](https://deployed-ai.io/blog/what-is-chunking/) matters.\n\n## ### Can text be recovered from its embedding?\n\nTo a large extent. Morris et al.'s vec2text (2023) reconstructed 92% of 32-token inputs exactly from their embeddings, for the encoder it was trained to invert, and recovered names from clinical notes. If the text is sensitive, the vectors are sensitive too: they need the same access controls, retention and deletion.", "url": "https://wpnews.pro/news/why-king-man-woman-isn-t-queen", "canonical_source": "https://deployed-ai.io/blog/what-is-an-embedding/", "published_at": "2026-11-04 09:50:55+00:00", "updated_at": "2026-11-04 10:24:52.062333+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-search", "natural-language-processing"], "entities": ["Qwen3-Embedding-0.6B", "Qwen", "word2vec", "gensim", "vec2text", "BM25"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-king-man-woman-isn-t-queen", "markdown": "https://wpnews.pro/news/why-king-man-woman-isn-t-queen.md", "text": "https://wpnews.pro/news/why-king-man-woman-isn-t-queen.txt", "jsonld": "https://wpnews.pro/news/why-king-man-woman-isn-t-queen.jsonld"}}