Hybrid Retrieval Under the Microscope: BM25 vs MiniCOIL on MedQuAD

A controlled experiment comparing hybrid retrieval pipelines using BM25 versus miniCOIL on the MedQuAD dataset with EmbeddingGemma and Qdrant shows that miniCOIL, a sparse neural retrieval model, improves contextual understanding over traditional BM25 by considering the meaning of matching terms rather than treating all occurrences equally. The benchmark isolates the impact of replacing BM25 with miniCOIL while keeping dense embeddings identical, aiming to evaluate retrieval accuracy for medical questions.

Today, we are going to learn how BM25 and miniCOIL perform in a hybrid search pipeline. Using the MedQuAD dataset, EmbeddingGemma, and Qdrant, we will build two retrieval setups, compare their benchmark results, and understand where miniCOIL improves over traditional BM25. BM25, or Best Matching 25 , is a keyword-based ranking algorithm used in search engines. It ranks documents by considering: For example, when searching for “vector database” , BM25 gives higher scores to documents containing both terms, especially when those words are uncommon and appear prominently. However, BM25 matches words statistically and does not understand their contextual meaning. miniCOIL is Qdrant’s open-source sparse neural retrieval model . It extends BM25-style keyword matching by considering the contextual meaning of matching words. For example, for the query “vectors in medicine” , BM25 may treat every occurrence of the word vector similarly. miniCOIL can understand that vector in “vector-borne diseases” is more relevant than vector in graphics or mathematics. miniCOIL is inspired by the COIL approach and can be understood as: BM25-style keyword retrieval with contextual understanding. Because miniCOIL still relies on overlapping words, it is commonly combined with dense embeddings in hybrid search to retrieve documents that use different terminology. For this experiment, I designed a controlled benchmarking architecture to compare two hybrid retrieval approaches in Qdrant. The first approach combines dense embeddings with BM25, while the second combines the same dense embeddings with miniCOIL. My intention was to keep every other part of the pipeline unchanged so that the final comparison would clearly show the impact of replacing BM25 with miniCOIL. The experiment begins with the MedQuAD dataset, which contains medical questions, answers, and their corresponding contexts. During ingestion, I use only the contextual content as the searchable knowledge base. The questions and expected answers are maintained separately for validation. This separation is important because the benchmark should evaluate whether the retrieval system can find the correct medical context for a given question, rather than simply matching against the question-answer pairs already stored in the database. The ingester processes each MedQuAD context and prepares it for storage in two separate Qdrant collections. For both collections, I generate dense vectors using the embeddinggemma-300m model. Using the same dense embedding model ensures that the semantic retrieval component remains identical across the two experiments. The only component that changes is the sparse retrieval method. The first Qdrant collection uses a combination of embeddinggemma-300m dense vectors and BM25 sparse vectors. BM25 represents the traditional lexical side of hybrid retrieval. It rewards documents that contain query terms based on factors such as term frequency, document length, and inverse document frequency. This collection therefore combines semantic similarity from the dense vectors with conventional keyword matching from BM25. The second collection uses the same embeddinggemma-300m dense vectors, but replaces BM25 with miniCOIL. Unlike traditional BM25, miniCOIL does not treat every occurrence of a matching word in exactly the same way. It generates contextualized sparse representations, allowing the retrieval system to consider the meaning of matching terms within the query and document. This makes the second collection an interesting alternative to the conventional dense-plus-BM25 setup. Once both collections are populated with the same MedQuAD contexts, the benchmark component reads questions from the MedQuAD validation dataset. Each validation question is submitted independently to both collections. The first retrieval run searches the dense-plus-BM25 collection, while the second searches the dense-plus-miniCOIL collection. Since both systems receive exactly the same questions and search over exactly the same content, the comparison remains fair and reproducible. For every query, the benchmark component records the ranked contexts returned by both retrieval configurations. These results are then compared with the expected context or answer associated with the validation question. The evaluation measures how often the correct context appears among the top retrieved results, how highly it is ranked, and how much time each retrieval approach requires. In simple terms, the architecture creates two parallel retrieval pipelines. Both pipelines use the same dataset, the same dense embedding model, the same Qdrant environment, and the same validation questions. The only deliberate difference is the sparse retrieval mechanism: BM25 in the first pipeline and miniCOIL in the second. This controlled setup allows me to determine whether miniCOIL provides a measurable improvement over BM25 when combined with dense retrieval for medical question answering. This script prepares the MedQuAD medical dataset for hybrid search in Qdrant using both dense embeddings and BM25 sparse vectors. It first downloads the dataset from Hugging Face and extracts only the medical context from each record. Every context is stored with a simple payload containing a document ID and the original text. Next, the script calculates the average document length, which BM25 uses during scoring. It then generates dense semantic embeddings for all contexts using google/embeddinggemma-300m. These dense vectors capture the meaning of the medical text, while BM25 captures exact keyword relevance. The script creates a Qdrant collection named medquad dense bm25 with two vector types: a dense vector using cosine similarity and a sparse BM25 vector using the IDF modifier. Each MedQuAD context is then stored as a Qdrant point containing the original text, its dense embedding, and its BM25 representation. The documents are uploaded to Qdrant in batches of 16. To make ingestion faster, indexing is temporarily delayed while the data is inserted. Once all documents are uploaded, the script triggers indexing and waits until the Qdrant collection reaches the green state. In simple terms, this code converts the MedQuAD contexts into a searchable Qdrant knowledge base that supports hybrid retrieval by combining semantic search from EmbeddingGemma with keyword search from BM25. python from datasets import load datasetfrom sentence transformers import SentenceTransformerfrom qdrant client import QdrantClient, models --- config ---HF REPO ID = "pavanmantha/MedQuAD Context GroupedQA"SPLIT = "train"MODEL NAME = "google/embeddinggemma-300m"BM25 MODEL = "Qdrant/bm25"COLLECTION NAME = "medquad dense bm25"QDRANT URL = "http://localhost:6333"API KEY = "API KEY"BATCH SIZE = 16RECREATE = Falsedef load contexts : """Pull the grouped dataset and return contexts, payloads .""" ds = load dataset HF REPO ID, split=SPLIT contexts, payloads = , for i, row in enumerate ds : ctx = row "context" contexts.append ctx payloads.append { "doc id": f"doc {i}", "text": ctx, } return contexts, payloadsdef compute avg length contexts : """Average word count across the corpus, part of BM25's scoring formula.""" return sum len c.split for c in contexts / len contexts def embed dense contexts : """Precompute dense EmbeddingGemma vectors, same as the original dense-only pipeline.""" print f"Embedding {len contexts } contexts with {MODEL NAME}..." model = SentenceTransformer MODEL NAME, device="mps" embeddings = model.encode document contexts, batch size=BATCH SIZE, show progress bar=True vectors = vec.tolist for vec in embeddings dim = len vectors 0 print f"Got {len vectors } dense vectors of dim {dim}" return vectors, dimdef create collection client, collection name, dense dim : if RECREATE and client.collection exists collection name : client.delete collection collection name if not client.collection exists collection name : client.create collection collection name=collection name, vectors config={ "dense": models.VectorParams size=dense dim, distance=models.Distance.COSINE, hnsw config=models.HnswConfigDiff m=64, ef construct=200, , , }, sparse vectors config={ "bm25": models.SparseVectorParams modifier=models.Modifier.IDF, , }, optimizers config=models.OptimizersConfigDiff indexing threshold=100000, , def build points contexts, payloads, dense vectors, avg len : points = for i, ctx, payload, dvec in enumerate zip contexts, payloads, dense vectors : points.append models.PointStruct id=i, payload=payload, vector={ "dense": dvec, "bm25": models.Document text=ctx, model=BM25 MODEL, options={"avg len": avg len}, , }, return pointsdef upsert points client, collection name, points : print "1. Upserting points dense + bm25 ..." for start in range 0, len points , BATCH SIZE : batch = points start:start + BATCH SIZE client.upsert collection name=collection name, points=batch print f" upserted {start + len batch }/{len points }" def trigger indexing client, collection name : print "2. All points upserted. Triggering indexing..." client.update collection collection name=collection name, optimizers config=models.OptimizersConfigDiff indexing threshold=1000, , import time while True: info = client.get collection collection name if info.status.value == "green": print "Index ready — collection is green" break print f" status: {info.status.value} — waiting..." time.sleep 5 print "3. indexing done" def main : contexts, payloads = load contexts avg len = compute avg length contexts dense vectors, dense dim = embed dense contexts client = QdrantClient url=QDRANT URL, api key=API KEY create collection client, COLLECTION NAME, dense dim points = build points contexts, payloads, dense vectors, avg len upsert points client, COLLECTION NAME, points trigger indexing client, COLLECTION NAME if name == " main ": main This script builds the second retrieval pipeline in the experiment, where dense semantic search is combined with miniCOIL instead of BM25. It downloads the MedQuAD dataset from Hugging Face, extracts the medical context from each record, and prepares the original text and document ID as payload data for Qdrant. The dense representation is generated using google/embeddinggemma-300m. To avoid recomputing the same embeddings every time the script runs, the vectors are saved in a local .cache folder using a pickle file. On later runs, the script checks whether the cached vectors match the current dataset size and loads them directly. This makes repeated ingestion much faster. Since both EmbeddingGemma and miniCOIL may consume significant memory, the script releases the dense embedding model after the vectors are generated. It clears Python memory and the Apple MPS cache before miniCOIL is used. This prevents memory conflicts when running both models in the same process on a Mac. The script calculates the average document length because miniCOIL uses a BM25-based scoring structure. It then creates a Qdrant collection called medquad dense minicoil with two named vector spaces. The dense vector stores the EmbeddingGemma representation and uses cosine similarity, while the minicoil vector stores the contextual sparse representation generated using Qdrant/minicoil-v1. The miniCOIL vector is also configured with the required IDF modifier. Each MedQuAD context is stored as a Qdrant point containing its payload, dense embedding, and miniCOIL representation. The documents are uploaded in batches of 16. Index construction is initially delayed to make ingestion faster, and after all the records are inserted, the indexing threshold is lowered so Qdrant can build the final search index. The script then waits until the collection reaches the green state, confirming that indexing has finished successfully. In simple terms, this code creates a searchable MedQuAD collection that combines semantic understanding from EmbeddingGemma with context-aware lexical matching from miniCOIL. This collection forms the miniCOIL side of the comparison against the dense-plus-BM25 pipeline. python import osos.environ.setdefault "TOKENIZERS PARALLELISM", "false" avoid HF tokenizer fork after MPS initimport gcimport picklefrom pathlib import Pathfrom datasets import load datasetfrom sentence transformers import SentenceTransformerfrom qdrant client import QdrantClient, models --- config ---HF REPO ID = "pavanmantha/MedQuAD Context GroupedQA"SPLIT = "train"MODEL NAME = "google/embeddinggemma-300m"MINICOIL MODEL = "Qdrant/minicoil-v1"COLLECTION NAME = "medquad dense minicoil"QDRANT URL = "http://localhost:6333"API KEY = "API key"BATCH SIZE = 16RECREATE = FalseCACHE DIR = Path file .resolve .parent / ".cache"CACHE DIR.mkdir exist ok=True print f"Cache dir: {CACHE DIR}" def load contexts : """Pull the grouped dataset and return contexts, payloads .""" ds = load dataset HF REPO ID, split=SPLIT contexts, payloads = , for i, row in enumerate ds : ctx = row "context" contexts.append ctx payloads.append { "doc id": f"doc {i}", "text": ctx, } return contexts, payloadsdef compute avg length contexts : """Average word count across the corpus, used by miniCOIL's BM25-based scoring.""" return sum len c.split for c in contexts / len contexts def embed dense contexts : """Precompute dense EmbeddingGemma vectors, cached to disk, model released from MPS before returning so it isn't still alive when minicoil's ONNX model loads during upsert.""" cache path = CACHE DIR / "dense.pkl" if cache path.exists : with open cache path, "rb" as f: cached len, vectors = pickle.load f if cached len == len contexts : print f"Loaded {cached len} cached dense vectors from {cache path}" return vectors, len vectors 0 print f"Cache is for {cached len} contexts, need {len contexts } — recomputing" print f"Embedding {len contexts } contexts with {MODEL NAME}..." model = SentenceTransformer MODEL NAME, device="mps" embeddings = model.encode document contexts, batch size=BATCH SIZE, show progress bar=True vectors = vec.tolist for vec in embeddings dim = len vectors 0 print f"Got {len vectors } dense vectors of dim {dim}" Release before minicoil's ONNX model ever loads in this same process del model gc.collect try: import torch torch.mps.empty cache except Exception: pass with open cache path, "wb" as f: pickle.dump len vectors , vectors , f print f"Cached dense vectors to {cache path}" return vectors, dimdef create collection client, collection name, dense dim : if RECREATE and client.collection exists collection name : client.delete collection collection name if not client.collection exists collection name : client.create collection collection name=collection name, vectors config={ "dense": models.VectorParams size=dense dim, distance=models.Distance.COSINE, hnsw config=models.HnswConfigDiff m=64, ef construct=200, , , }, sparse vectors config={ "minicoil": models.SparseVectorParams modifier=models.Modifier.IDF, , }, optimizers config=models.OptimizersConfigDiff indexing threshold=100000, , def build points contexts, payloads, dense vectors, avg len : points = for i, ctx, payload, dvec in enumerate zip contexts, payloads, dense vectors : points.append models.PointStruct id=i, payload=payload, vector={ "dense": dvec, "minicoil": models.Document text=ctx, model=MINICOIL MODEL, options={"avg len": avg len}, , }, return pointsdef upsert points client, collection name, points : print "1. Upserting points dense + minicoil ..." for start in range 0, len points , BATCH SIZE : batch = points start:start + BATCH SIZE client.upsert collection name=collection name, points=batch print f" upserted {start + len batch }/{len points }" def trigger indexing client, collection name : print "2. All points upserted. Triggering indexing..." client.update collection collection name=collection name, optimizers config=models.OptimizersConfigDiff indexing threshold=1000, , import time while True: info = client.get collection collection name if info.status.value == "green": print "Index ready — collection is green" break print f" status: {info.status.value} — waiting..." time.sleep 5 print "3. indexing done" def main : contexts, payloads = load contexts avg len = compute avg length contexts dense vectors, dense dim = embed dense contexts client = QdrantClient url=QDRANT URL, api key=API KEY create collection client, COLLECTION NAME, dense dim points = build points contexts, payloads, dense vectors, avg len upsert points client, COLLECTION NAME, points trigger indexing client, COLLECTION NAME if name == " main ": main These benchmark results make the comparison much clearer. Both configurations successfully evaluated all 100 questions without errors. The dense-plus-BM25 pipeline achieved a contextual precision of 0.879 and contextual recall of 0.840 , while the dense-plus-miniCOIL pipeline improved these scores to 0.898 and 0.875 , respectively. In practical terms, miniCOIL improved contextual precision by approximately 1.93 percentage points and contextual recall by 3.51 percentage points . The larger improvement in recall suggests that miniCOIL was more successful at retrieving the complete set of medical information required to answer the validation questions. Its contextualized term representations appear to provide an advantage over traditional BM25 when matching specialised medical queries with relevant contexts. Answer relevancy remained almost identical across both pipelines. BM25 achieved 0.3997 , while miniCOIL achieved 0.3990 . This difference is negligible and indicates that changing the sparse retrieval model improved retrieval quality but did not improve how the generation model used the retrieved information. The answer-generation stage remains a separate area that requires further optimisation. { "bm25": { "n scored": 100, "n errors": 0, "avg contextual precision": 0.8785067433768732, "avg contextual recall": 0.8398373220591189, "avg answer relevancy": 0.39968247900511716 }, "minicoil": { "n scored": 100, "n errors": 0, "avg contextual precision": 0.8978049792139078, "avg contextual recall": 0.8749693639693639, "avg answer relevancy": 0.3989726123549653 }} In this article, we learned what BM25 is, how miniCOIL extends lexical retrieval with contextual understanding, and how both approaches can be combined with dense embeddings to build hybrid search pipelines in Qdrant. We designed an experimental architecture using the MedQuAD dataset, EmbeddingGemma for dense representations, and two separate Qdrant collections to compare dense retrieval combined with BM25 and miniCOIL. We also implemented the complete workflow, from loading and preparing the dataset to generating vectors, ingesting them into Qdrant, executing validation queries, and evaluating the retrieved results. By keeping the dataset, dense embedding model, validation questions, and Qdrant configuration consistent, the experiment created a fair environment for studying the effect of changing only the sparse retrieval model. The benchmark produced strong retrieval results, with an average contextual precision of 0.879 and contextual recall of 0.840 . These scores show that the system was generally successful in retrieving and ranking the medical contexts required to answer the validation questions. However, the lower answer relevancy score also highlighted an important lesson: retrieving the correct context does not automatically guarantee a correct final answer. The generation model must still understand, follow, and accurately use the retrieved evidence. Overall, BM25 remains a reliable and efficient choice when exact keyword matching is important, while miniCOIL introduces contextual awareness into sparse retrieval and can better distinguish the meaning of matching terms. The right choice depends on the dataset, query patterns, latency requirements, and domain. Rather than selecting a retrieval technique based only on its popularity, this experiment shows why it is important to benchmark each configuration using real domain data and measurable evaluation metrics. Hybrid Retrieval Under the Microscope: BM25 vs MiniCOIL on MedQuAD https://blog.stackademic.com/hybrid-retrieval-under-the-microscope-bm25-vs-minicoil-on-medquad-129b464b0166 was originally published in Stackademic https://blog.stackademic.com on Medium, where people are continuing the conversation by highlighting and responding to this story.