Combine dense embeddings and sparse keyword search using Reciprocal Rank Fusion to eliminate retrieval failure modes in production RAG systems.
Most production RAG pipelines start with pure vector search. It works reliably during initial demos, but fails quietly once users begin searching for exact product IDs, error codes, or domain-specific identifiers.
Vector search translates text into semantic coordinate spaces. Because dense models optimize for high-level concepts, they tend to blur fine-grained details. A search for ERR-502-BAD-GATEWAY
might retrieve general network troubleshooting docs instead of the exact runbook for error 502
.
Conversely, relying purely on keyword search (BM25) breaks when users paraphrase. A query for "reduce database memory footprint" completely misses an article titled "Mitigating PostgreSQL RAM Saturation" if there is no direct keyword overlap.
def get_context(query: str, vector_db) -> list[str]:
return vector_db.similarity_search(query, k=5)
Relying on a single retrieval strategy creates hard blind spots that degrade downstream LLM generation quality.
The fix is running sparse (BM25) and dense (vector) searches concurrently and merging their ranked outputs using Reciprocal Rank Fusion (RRF).
┌───> [ BM25 Keyword Search ] ───> Sparse Ranked List ───┐
[ User Query ] ──────┤ ├───> [ RRF Fusion ] ───> Top-K Documents ───> LLM
└───> [ Dense Vector Search ] ───> Dense Ranked List ───┘
RRF avoids the core challenge of hybrid search: score normalization. BM25 produces unbounded positive floating-point scores, while vector search usually outputs cosine similarities between -1
and 1
. Merging raw scores requires fragile heuristic scaling factors.
Instead of comparing raw scores, RRF scores documents based strictly on their relative rank position across both search lists:
$$RRF(d) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$
Where:
60
) that prevents low-ranking outliers from disproportionately skewing results.Here is a minimal, production-ready implementation combining BM25, dense embeddings, and RRF in under 25 lines of Python:
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
def hybrid_rrf_search(query: str, corpus: list[str], top_n: int = 3, k: int = 60) -> list[str]:
bm25 = BM25Okapi([doc.lower().split() for doc in corpus])
bm25_rank = np.argsort(bm25.get_scores(query.lower().split()))[::-1]
model = SentenceTransformer('all-MiniLM-L6-v2')
doc_embs, query_emb = model.encode(corpus), model.encode([query])
dense_rank = np.argsort(cosine_similarity(query_emb, doc_embs)[0])[::-1]
rrf_scores = {}
for rank_idx, doc_idx in enumerate(bm25_rank):
rrf_scores[doc_idx] = rrf_scores.get(doc_idx, 0.0) + (1.0 / (k + rank_idx + 1))
for rank_idx, doc_idx in enumerate(dense_rank):
rrf_scores[doc_idx] = rrf_scores.get(doc_idx, 0.0) + (1.0 / (k + rank_idx + 1))
sorted_docs = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
return [corpus[doc_idx] for doc_idx, _ in sorted_docs[:top_n]]
This pattern guarantees resilience. If a user inputs a technical keyword match, BM25 pushes the correct document to the top. If a user describes a high-level conceptual problem, dense