The worst retrieval bug I know of throws no error. Latency stays flat. The error rate sits at zero. The index reports healthy, and every query still returns ten results.
Some of those results are wrong, and nothing in your stack can tell you which ones.
That is what a bad embedding model upgrade looks like in production. To write this I read vendor docs, engineering write ups, a developer forum thread and one EMNLP paper. They agree on one point: the failure is quiet by design. Below are the three facts behind it, five paths into it, and the checks that make it loud.
An embedding model upgrade breaks retrieval when stored document vectors and fresh query vectors come from different models. Each model maps text into its own coordinate space, so similarity between the two is meaningless. The vector database still returns nearest neighbors, so nothing errors, and answers quietly get worse.
Three facts explain the whole problem:
Fix the third fact first. You cannot protect a system you do not measure.
Zilliz’s engineering team states it directly: a vector for the same document from version one of a model and version two of that model sit in different spaces, so cosine similarity between them has no meaning. Their guidance covers Milvus, Pinecone, Qdrant and Weaviate alike.
Picture two maps of one city, each drawn with a different origin and rotation. Both maps are accurate. Copy a pin from one onto the other and it lands in the wrong neighborhood.
An index holding both kinds of vectors behaves the same way. Zilliz notes that mixing versions in one HNSW or IVF index returns some neighbors from a completely wrong neighborhood. The database cannot see this. It only sees numbers of the right length.
Some upgrades announce themselves. Google moved developers from text-embedding-004 to gemini-embedding-001, and the vector size jumped from 768 to 3072. In a public n8n community thread, inserting the new vectors into a 768 dimension store returned a 400 error that named the mismatch.
That error is a gift. Someone files a ticket and the team fixes it the same day.
The same thread offers a workaround: OpenAI’s text-embedding-3-small can be configured to return 768 dimensions. That is fine if you reembed every document with the new model. If you keep the old vectors, matching the size has only matched the shape of the data. The spaces stay misaligned and the alarm is gone.
Same size, different space is the case to fear. The index accepts every write, every query runs, and the only symptom is relevance.
You change the model name in config and the vector size stays the same. Existing vectors stay in the index. Every new query is encoded in the new space and compared against documents from the old one.
Tian Pan’s write up on embedding model churn describes providers that keep the right to upgrade or swap the underlying model without changing the endpoint name or giving notice. Most API terms permit it. You changed nothing, and your index is still invalidated.
The same write up calls in place replacement the most dangerous strategy. You stop writes, overwrite vectors, and resume, which leaves a window where the index mixes old and new vectors. Mixpeek’s glossary adds that large jobs that encode the whole corpus again will fail partway through. A job that dies midway leaves the mix in place until someone notices.
A shared client library bumps its default model, for example. Queries now come from the new encoder while the corpus still holds old vectors. Tian Pan describes exactly this mismatch: an index full of vectors from a coordinate system that no longer matches what the query encoder produces.
A Prompt/Deploy breakdown of reindex triggers says a model change needs a full reindex, a chunking change needs a full reindex, and doing both together cannot be separated because both are tied to the vectors. If quality drops, you cannot tell which change caused it. Ship one variable per release.
Approximate nearest neighbor search has no concept of “no good answer.” It returns the closest points it can find. A broken index returns ten results with similarity scores that look like any other numbers.
Your retriever hands those passages to the language model. The model writes a fluent answer from the wrong material. The incident now reads as a hallucination, and the team starts tuning prompts for a problem that lives in the data layer.
The cheapest detector is a self consistency test. Take a random sample of stored documents, embed them again with the model your query path uses today, and compare the fresh vector to the stored one. If nothing changed, the two should be nearly identical.
python
import randomimport numpy as npdef cosine(a, b): a, b = np.asarray(a), np.asarray(b) return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))def drift_check(store, embed, sample_size=200, seed=7): # store.all_ids(), store.get_text(id), store.get_vector(id) are your own accessors. # embed(text) calls the same model your query path calls today. ids = list(store.all_ids()) random.Random(seed).shuffle(ids) scores = [] for doc_id in ids[:sample_size]: stored = store.get_vector(doc_id) fresh = embed(store.get_text(doc_id)) scores.append(cosine(stored, fresh)) return { "mean": float(np.mean(scores)), "p5": float(np.percentile(scores, 5)), "min": float(np.min(scores)), }
Treat the numbers carefully. Some providers return tiny differences between identical calls, so the right alarm threshold depends on your stack. Run the check on a day when you know nothing changed, record that baseline, and alert when the mean or the fifth percentile falls below it.
Schedule it. A provider swap on a Tuesday should not wait for a quarterly review to be found.
Drift tells you the vectors changed. A canary set tells you whether users feel it. Collect real queries from your logs, label which documents should come back for each, and track how often at least one correct document lands in the top 10.
def hit_rate_at_k(search, golden, k=10): # golden is a list of (query, set_of_relevant_doc_ids) hits = 0 for query, relevant_ids in golden: results = set(search(query, k)) if results & set(relevant_ids): hits += 1 return hits / len(golden)
I would start with 50 to 100 queries and grow the set whenever a user reports a bad answer. Run it before and after every change to the embedding path. A migration without this gate is a guess.
These six steps follow the pattern the sources above converge on.
Not every change needs the full treatment. The Prompt/Deploy breakdown draws the line this way:
ChangeReindex needed?New metadata field, no content changeUsually no, upsert metadata onlyDocument content updatePartial, changed documents onlyChunking strategy changeYes, full reindexEmbedding model changeYes, full reindexModel and chunking change togetherYes, full reindex, and the effects are inseparable
Mixpeek gives a useful anchor: a corpus of 100 million documents at $0.0001 per embedding costs $10,000 and several days of compute.
That bill is real. It is also small next to a quarter of degraded search that nobody measured. High cost is the reason teams postpone migrations until a deprecation deadline forces one, which Mixpeek lists as a common pitfall because it leaves no room for gradual rollout or validation.
Researchers are working on ways to skip the full reembed. The Drift Adapter paper, published in EMNLP 2025 main, trains a small transformation that maps new queries into the legacy embedding space, so the existing index keeps serving. On MTEB text corpora and a CLIP image model upgrade covering 1 million items, it recovered 95 to 99 percent of the recall of a full reembed while adding only microseconds of query latency. (arXiv)
Tian Pan’s write up also points to Vec2Vec style translation between model spaces, reporting cosine similarity above 0.9 to native embeddings across architectures.
These are promising bridges. The paper itself frames the adapter as a way to defer full recomputation. Read the numbers as a loss of 1 to 5 percent of recall against a full reembed. For a contract search or compliance tool, that remainder is where the missed clause lives. Use an adapter to buy time, then finish the migration.
Four steps, one afternoon, and the next silent break becomes a visible alert.
No. Each model creates its own coordinate space, so similarity between vectors from different models is meaningless. A mixed index returns some neighbors from the wrong neighborhood without raising an error.
Embed a random sample of stored documents again and compare the fresh vectors to the stored ones. A drop below your recorded baseline signals drift. Pair it with a canary query set so you see the effect on real retrieval.
For a model change or a chunking change, yes, a full reembed is needed. Metadata only changes usually need no reembedding, and content edits need only the changed documents. Adapter research can defer the full job but does not remove it.
Tian Pan suggests keeping it until you have two to four weeks of post migration signal. That window lets you roll back if a quality regression shows up late.
Not entirely. Drift Adapter recovered 95 to 99 percent of full reembed recall in its experiments. Treat it as a bridge that buys time while you run the real migration.
Treat an embedding model upgrade as a data migration. Version your vectors, measure drift on a schedule, gate cutover on a canary set, and keep the old index for a few weeks.
If this saved you from a silent break, follow The AI Engineer for more production failure analysis. And if you have already been bitten, tell me where it broke in the comments.
Why Your Embeddings Model Upgrade Silently Broke Retrieval was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.