A user typed E4012 into my docs assistant. The top result was the page for E4021. The second was E4013. The page that actually explained E4012 sat at rank 14, well below my top-5 cutoff, so the LLM never saw it and confidently explained the wrong error.
That's the BM25 vs embeddings problem in one screenshot. Dense vector search is great at "what does this mean" and bad at "find this exact string." Error codes, SKUs, function names, ticket IDs and version numbers fall straight into that gap.
I spent an evening convinced my index was corrupted. It wasn't. The embedding model was doing exactly what it was built to do.
E4012 and E4021 mean almost the same thing to them ("some error code in this product"), so their vectors land next to each other.ERR_CONN_RESET or v2.3.1 into pieces throws away the exact-match advantage you added BM25 for.
Vector search fails on identifiers because an embedding encodes what a passage is about, not which exact characters it contains. Two pages that describe neighboring error codes are about the same thing, so they get nearly the same vector.
Walk through what happens to E4012:
E, 40, 12. None of those fragments is special. 21. Most of the pieces overlap.
The result: the E4012 page and the E4021 page sit a hair apart in vector space, and which one wins for the query E4012 is basically noise.
A short query makes it worse. E4012 alone has almost no context, so its embedding is a vague "error-code-ish" vector. It's closer to every error page than to any particular one.
BM25 ranks documents by counting exact term matches and weighting each term by how rare it is across the corpus. A term that appears in only one document carries a very high weight, so that document wins.
The scoring formula has three moving parts:
log((N - n + 0.5) / (n + 0.5) + 1), where N is the number of docs and n is how many contain the term. In my corpus of about 2,300 chunks, a word like "connection" appeared in hundreds of chunks. e4012 appeared in 3.k1 parameter (default around 1.2 to 1.5) means mentioning the term 10 times isn't 10 times better than mentioning it once.b parameter (default 0.75) stops long chunks from winning just by being long.
Plug in the numbers and e4012 gets an IDF around 6.5 while "connection" sits near 1.5. BM25 doesn't know what an error code is. It just knows this token is rare and the query contains it. For identifier lookups, that's exactly the right instinct.
Here's the minimal version with rank_bm25:
import re
from rank_bm25 import BM25Okapi
def tokenize(text: str) -> list[str]:
return re.findall(r"[A-Za-z0-9_][A-Za-z0-9_.\-]*[A-Za-z0-9_]|[A-Za-z0-9]", text.lower())
corpus_tokens = [tokenize(c.text) for c in chunks]
bm25 = BM25Okapi(corpus_tokens, k1=1.2, b=0.75)
scores = bm25.get_scores(tokenize("E4012"))
My first BM25 attempt failed because the tokenizer split identifiers into pieces before indexing them. Once ERR_CONN_RESET becomes err, conn, reset, three common words, the rare-term advantage is gone.
I started with text.split(). That kept E4012, with the trailing comma as a different token from E4012, so half my matches silently vanished. Then I switched to a "proper" word tokenizer that stripped punctuation, and it shredded v2.3.1 into v2, 3, 1 and ERR_CONN_RESET into three generic words.
Things to check in whatever BM25 you use (Elasticsearch, OpenSearch, Postgres full-text, Tantivy, rank_bm25):
_, ., or -? Paste one of your real identifiers through it and look. errors into error is fine. One that mangles HTTP_403s is not.
The regex above is crude, and that's the point. It keeps anything that looks like an identifier in one piece. Run your nastiest IDs through it before trusting it.
You combine them by running both searches and merging the two ranked lists with Reciprocal Rank Fusion. RRF ignores raw scores and only uses rank positions, which matters because BM25 scores and cosine similarities live on completely different scales.
The formula: for each document, sum 1 / (k + rank) across every list it appears in. k = 60 is the common default from the original RRF paper.
def rrf(result_lists: list[list[str]], k: int = 60, top_n: int = 5) -> list[str]:
scores: dict[str, float] = {}
for results in result_lists:
for rank, doc_id in enumerate(results, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)[:top_n]
dense_ids = vector_search(query, limit=50)
sparse_ids = bm25_search(query, limit=50)
context_ids = rrf([dense_ids, sparse_ids])
Why this works for E4012: BM25 puts the right page at rank 1 (score 1/61). Dense search had it at rank 14 ( 1/74). Total about 0.030. The E4021 page was dense rank 1 but didn't appear in BM25's list at all, so it scores 1/61 or about 0.016. The right page wins by almost double.
And for a fuzzy question like "why does my upload hang forever," BM25 returns weak, scattered matches while dense search nails it. RRF lets the stronger signal win each time without you writing a router.
Pull 50 candidates from each side, not 5. If the right doc is rank 14 in one list and you only fetch 10, fusion never sees it.
On my own test set, hybrid search recovered every identifier query that dense-only search missed and left natural-language queries roughly where they were. These are numbers from my corpus and my embedding model, not a general benchmark.
The setup: 120 real queries pulled from support tickets, each labeled with the chunk that answers it. 38 of them contained an identifier (error code, config key, CLI flag, version string). I measured whether the correct chunk landed in the top 5.
120 queries is small. Treat it as "this is the direction" rather than "this is the number." Build your own eval set from real queries; it's the only one that tells you anything about your corpus.
A cross-encoder reranker helps ordering, but it can only rerank what retrieval hands it. If dense search buried the right chunk at rank 14 and you fetched 10 candidates, no reranker will ever see it.
Rerankers also read the same subword fragments, so they're better at identifiers than bi-encoders but not immune. Put the reranker after hybrid retrieval, not instead of it.
[A-Z]+\d{3,}, \w+_\w+, or v\d+\.\d+. That's my identifier traffic share. Mine was about a third.
Use both. Embeddings find passages that mean the same thing as the query; BM25 finds passages that contain the same rare tokens. Vector search misses error codes like E4012 because embedding models compress the code's digits into a vector dominated by surrounding meaning, which puts neighboring codes almost on top of each other. BM25 gives that rare exact token a high IDF weight and ranks the matching document first. Run both retrievers, fetch around 50 candidates from each, merge with Reciprocal Rank Fusion at k = 60, and make sure your BM25 tokenizer keeps identifiers in one piece. That combination handles both "what does this error mean" and "find error E4012."
Written by the developer behind Preterview, an interview prep platform.