{"slug": "bm25-vs-embeddings-why-my-vector-search-couldn-t-find-error-e4012", "title": "BM25 vs Embeddings: Why My Vector Search Couldn't Find Error E4012", "summary": "A developer documented why dense vector search failed to surface the correct documentation page for the error code E4012, ranking it 14th while near-miss codes like E4021 and E4013 took the top slots. The engineer traced the failure to embeddings encoding what a passage is about rather than which exact characters it contains, and showed that BM25 with an identifier-preserving tokenizer gives rare tokens like e4012 an IDF around 6.5 versus roughly 1.5 for a common word like \"connection\" in a corpus of about 2,300 chunks. The recommended fix is running BM25 and vector search in parallel and merging results with Reciprocal Rank Fusion.", "body_md": "A user typed `E4012` into my docs assistant. The top result was the page for `E4021`. The second was `E4013`. The page that actually explained `E4012` sat at rank 14, well below my top-5 cutoff, so the LLM never saw it and confidently explained the wrong error.\n\nThat's the **BM25 vs embeddings** problem in one screenshot. Dense vector search is great at \"what does this mean\" and bad at \"find this exact string.\" Error codes, SKUs, function names, ticket IDs and version numbers fall straight into that gap.\n\nI spent an evening convinced my index was corrupted. It wasn't. The embedding model was doing exactly what it was built to do.\n\n`E4012` and `E4021` mean almost the same thing to them (\"some error code in this product\"), so their vectors land next to each other.`ERR_CONN_RESET` or `v2.3.1` into pieces throws away the exact-match advantage you added BM25 for.\nVector search fails on identifiers because an embedding encodes what a passage is *about*, not which exact characters it contains. Two pages that describe neighboring error codes are about the same thing, so they get nearly the same vector.\n\nWalk through what happens to `E4012`:\n\n`E`, `40`, `12`. None of those fragments is special. `21`. Most of the pieces overlap.\nThe result: the `E4012` page and the `E4021` page sit a hair apart in vector space, and which one wins for the query `E4012` is basically noise.\n\nA short query makes it worse. `E4012` alone has almost no context, so its embedding is a vague \"error-code-ish\" vector. It's closer to *every* error page than to any particular one.\n\nBM25 ranks documents by counting exact term matches and weighting each term by how rare it is across the corpus. A term that appears in only one document carries a very high weight, so that document wins.\n\nThe scoring formula has three moving parts:\n\n`log((N - n + 0.5) / (n + 0.5) + 1)`, where `N` is the number of docs and `n` is how many contain the term. In my corpus of about 2,300 chunks, a word like \"connection\" appeared in hundreds of chunks. `e4012` appeared in 3.`k1` parameter (default around 1.2 to 1.5) means mentioning the term 10 times isn't 10 times better than mentioning it once.`b` parameter (default 0.75) stops long chunks from winning just by being long.\nPlug in the numbers and `e4012` gets an IDF around 6.5 while \"connection\" sits near 1.5. BM25 doesn't know what an error code *is*. It just knows this token is rare and the query contains it. For identifier lookups, that's exactly the right instinct.\n\nHere's the minimal version with `rank_bm25`:\n\n``` php\nimport re\nfrom rank_bm25 import BM25Okapi\n\ndef tokenize(text: str) -> list[str]:\n    # keep codes like E4012, ERR_CONN_RESET, v2.3.1 as single tokens\n    return re.findall(r\"[A-Za-z0-9_][A-Za-z0-9_.\\-]*[A-Za-z0-9_]|[A-Za-z0-9]\", text.lower())\n\ncorpus_tokens = [tokenize(c.text) for c in chunks]\nbm25 = BM25Okapi(corpus_tokens, k1=1.2, b=0.75)\n\nscores = bm25.get_scores(tokenize(\"E4012\"))\n```\n\nMy first BM25 attempt failed because the tokenizer split identifiers into pieces before indexing them. Once `ERR_CONN_RESET` becomes `err`, `conn`, `reset`, three common words, the rare-term advantage is gone.\n\nI started with `text.split()`. That kept `E4012,` with the trailing comma as a different token from `E4012`, so half my matches silently vanished. Then I switched to a \"proper\" word tokenizer that stripped punctuation, and it shredded `v2.3.1` into `v2`, `3`, `1` and `ERR_CONN_RESET` into three generic words.\n\nThings to check in whatever BM25 you use (Elasticsearch, OpenSearch, Postgres full-text, Tantivy, `rank_bm25`):\n\n`_`, `.`, or `-`? Paste one of your real identifiers through it and look.` errors` into `error` is fine. One that mangles `HTTP_403s` is not.\nThe regex above is crude, and that's the point. It keeps anything that looks like an identifier in one piece. Run your nastiest IDs through it before trusting it.\n\nYou combine them by running both searches and merging the two ranked lists with Reciprocal Rank Fusion. RRF ignores raw scores and only uses rank positions, which matters because BM25 scores and cosine similarities live on completely different scales.\n\nThe formula: for each document, sum `1 / (k + rank)` across every list it appears in. `k = 60` is the common default from the original RRF paper.\n\n``` php\ndef rrf(result_lists: list[list[str]], k: int = 60, top_n: int = 5) -> list[str]:\n    scores: dict[str, float] = {}\n    for results in result_lists:\n        for rank, doc_id in enumerate(results, start=1):\n            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)\n    return sorted(scores, key=scores.get, reverse=True)[:top_n]\n\ndense_ids = vector_search(query, limit=50)\nsparse_ids = bm25_search(query, limit=50)\ncontext_ids = rrf([dense_ids, sparse_ids])\n```\n\nWhy this works for `E4012`: BM25 puts the right page at rank 1 (score `1/61`). Dense search had it at rank 14 (` 1/74`). Total about `0.030`. The `E4021` page was dense rank 1 but didn't appear in BM25's list at all, so it scores `1/61` or about `0.016`. The right page wins by almost double.\n\nAnd for a fuzzy question like \"why does my upload hang forever,\" BM25 returns weak, scattered matches while dense search nails it. RRF lets the stronger signal win each time without you writing a router.\n\nPull 50 candidates from each side, not 5. If the right doc is rank 14 in one list and you only fetch 10, fusion never sees it.\n\nOn my own test set, hybrid search recovered every identifier query that dense-only search missed and left natural-language queries roughly where they were. These are numbers from my corpus and my embedding model, not a general benchmark.\n\nThe setup: 120 real queries pulled from support tickets, each labeled with the chunk that answers it. 38 of them contained an identifier (error code, config key, CLI flag, version string). I measured whether the correct chunk landed in the top 5.\n\n120 queries is small. Treat it as \"this is the direction\" rather than \"this is the number.\" Build your own eval set from real queries; it's the only one that tells you anything about your corpus.\n\nA cross-encoder reranker helps ordering, but it can only rerank what retrieval hands it. If dense search buried the right chunk at rank 14 and you fetched 10 candidates, no reranker will ever see it.\n\nRerankers also read the same subword fragments, so they're better at identifiers than bi-encoders but not immune. Put the reranker after hybrid retrieval, not instead of it.\n\n`[A-Z]+\\d{3,}`, `\\w+_\\w+`, or `v\\d+\\.\\d+`. That's my identifier traffic share. Mine was about a third.\nUse both. Embeddings find passages that mean the same thing as the query; BM25 finds passages that contain the same rare tokens. Vector search misses error codes like `E4012` because embedding models compress the code's digits into a vector dominated by surrounding meaning, which puts neighboring codes almost on top of each other. BM25 gives that rare exact token a high IDF weight and ranks the matching document first. Run both retrievers, fetch around 50 candidates from each, merge with Reciprocal Rank Fusion at `k = 60`, and make sure your BM25 tokenizer keeps identifiers in one piece. That combination handles both \"what does this error mean\" and \"find error E4012.\"\n\n*Written by the developer behind [Preterview](https://preterview.com/en), an interview prep platform.*", "url": "https://wpnews.pro/news/bm25-vs-embeddings-why-my-vector-search-couldn-t-find-error-e4012", "canonical_source": "https://dev.to/ji_ai/bm25-vs-embeddings-why-my-vector-search-couldnt-find-error-e4012-9bi", "published_at": "2026-10-19 23:30:03+00:00", "updated_at": "2026-10-19 23:45:44.652767+00:00", "lang": "en", "topics": ["ai-search", "ai-tools", "natural-language-processing", "artificial-intelligence"], "entities": ["rank_bm25", "BM25Okapi", "Elasticsearch", "OpenSearch", "Tantivy", "Postgres"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/bm25-vs-embeddings-why-my-vector-search-couldn-t-find-error-e4012", "markdown": "https://wpnews.pro/news/bm25-vs-embeddings-why-my-vector-search-couldn-t-find-error-e4012.md", "text": "https://wpnews.pro/news/bm25-vs-embeddings-why-my-vector-search-couldn-t-find-error-e4012.txt", "jsonld": "https://wpnews.pro/news/bm25-vs-embeddings-why-my-vector-search-couldn-t-find-error-e4012.jsonld"}}