{"slug": "hybrid-retrieval-v2-qwen-embeddings-bm25-and-rrf-with-a-fastembed-reranker", "title": "Hybrid Retrieval v2: Qwen Embeddings, BM25, and RRF with a FastEmbed Reranker", "summary": "A developer overhauled their hybrid retrieval pipeline for technical documentation, combining Qwen embeddings, BM25 sparse vectors, and reciprocal rank fusion with a FastEmbed reranker. The upgrade improved retrieval accuracy from 5 to 8 correct top-3 results on a 10-query eval set, addressing the failure of dense-only retrieval on exact identifiers like config keys and error strings. The developer also documented a silent failure mode when serving cross-encoder rerankers through Ollama, where GGUF conversions can return untrained logits that degrade ranking quality.", "body_md": "A query for `ndots:5`\n\nagainst my wiki index used to return the article that exists specifically to explain `ndots:5`\n\nat position seven. Ahead of it sat three general DNS articles, two Kubernetes networking posts, and something about service discovery. My embedding model understood the *topic* perfectly and had no idea that the literal string mattered.\n\nThat is the dense retrieval failure mode in one sentence. Semantic similarity is a fuzzy match by design, and a fuzzy match is exactly wrong when the user typed an exact identifier. Across a fixed 10-query eval set of the things I actually search for (config keys, error strings, CLI flags), dense-only retrieval put the correct article in the top 3 for 5 of them. Hybrid retrieval with a reranker on the same 10 queries hits 8.\n\nIf you run a retrieval layer for agents and your corpus is technical documentation, code, runbooks, or session memories, you have this problem whether or not you've measured it. Technical corpora are full of tokens that carry near-zero semantic weight and near-total discriminative weight: `max_cstate`\n\n, `Modifier.IDF`\n\n, `ErrImagePull`\n\n, a CVE number, a Helm value path. An embedding model compresses all of those into a vector where they barely register against the surrounding prose.\n\nMy first instinct was to reach for a better embedding model. That instinct was wrong, and the reason it was wrong is the most useful thing in this post.\n\n**Bigger embeddings.** Swapping to a larger dense model moved my eval by roughly one query out of ten, and cost more VRAM plus more latency per ingest batch. Larger dense models are better at nuance in prose. None of them are better at treating `ndots:5`\n\nas an atomic symbol, because none of them are trained to. Adding dimensions does not create a keyword index.\n\n**Query expansion with an LLM.** Rewrite the user query into three paraphrases, embed all three, union the results. This helped on vague questions and actively hurt on precise ones, because the paraphrases diluted the exact term being searched for. It also adds an LLM round trip to every retrieval call, which turns a 40ms operation into a 900ms one and makes results non-deterministic between runs. Acceptable for a chat UI. Bad for an agent that retrieves twenty times inside a single task.\n\n**A cross-encoder reranker served through Ollama.** This one is worth writing down, because the failure was silent and cost the most time.\n\nMy plan was reasonable: over-retrieve 20 candidates from dense search, then rerank with a cross-encoder that sees query and document together. Ollama was already running in the cluster, GGUF conversions of popular rerankers exist on Hugging Face, so pull one, hit the API, sort by score.\n\nScores came back as numbers. They were garbage. Not obviously broken (no errors, no NaNs), just weakly correlated with relevance. Sometimes the reranked order was measurably worse than the pre-rerank order, which is an impressive achievement for a component whose entire job is to improve ordering.\n\nHere's the mechanism. A cross-encoder reranker is a sequence-classification model: an encoder backbone plus a trained classification head that emits a single relevance logit. Convert that to GGUF, serve it through a runtime built for causal LM generation and embedding extraction, and the classification head is usually not part of the picture. What comes back is a pooled hidden state, or a logit from a head that was never trained for relevance ranking, wrapped in a response shape identical to a real score. Nothing warns you. Your pipeline runs, your latency budget looks fine, and retrieval quality quietly rots.\n\nGeneralizing: when a model's output is a scalar, you cannot tell by inspection whether it's the *right* scalar. Test rerankers against a fixed query set with known-correct answers before you wire them in, not after you've shipped them.\n\nFour pieces. A Qdrant collection with two named vector spaces, dense embeddings from `qwen3-embedding:0.6b`\n\n, sparse BM25 vectors, and a cross-encoder reranker running on ONNX through FastEmbed. No GPU is involved in the reranking stage at all.\n\nDense and sparse vectors live on the *same point*. One document, one ID, two vector representations, one payload. That detail matters more than it looks: split them across two collections and you get two ingest paths that drift out of sync, and you'll find out about the drift during a retrieval failure at the worst possible moment.\n\n``` python\nfrom qdrant_client import QdrantClient, models\n\nclient = QdrantClient(url=\"http://qdrant:6333\")\n\nclient.create_collection(\n    collection_name=\"wiki_index_v2\",\n    vectors_config={\n        \"dense\": models.VectorParams(\n            size=1024,                      # qwen3-embedding:0.6b\n            distance=models.Distance.COSINE,\n        ),\n    },\n    sparse_vectors_config={\n        \"bm25\": models.SparseVectorParams(\n            # Qdrant applies IDF server-side against the live corpus\n            modifier=models.Modifier.IDF,\n        ),\n    },\n)\n```\n\n`modifier=models.Modifier.IDF`\n\nis the line people skip. FastEmbed's BM25 produces the term-frequency component client-side, but inverse document frequency depends on the entire corpus, and your corpus changes on every ingest. Setting the modifier makes Qdrant compute IDF at query time from current collection statistics. Leave it out and you're doing raw term-frequency matching, which over-weights common tokens and makes the sparse leg noticeably worse: in my eval it cost two of the eight top-3 hits.\n\nYou cannot add the modifier later without recreating the collection. Set it on day one.\n\n``` python\nfrom fastembed import SparseTextEmbedding\nimport ollama\n\nbm25 = SparseTextEmbedding(\"Qdrant/bm25\")\n\ndef embed_dense(text: str) -> list[float]:\n    return ollama.embed(model=\"qwen3-embedding:0.6b\", input=text)[\"embeddings\"][0]\n\ndef to_point(doc_id: int, text: str, payload: dict) -> models.PointStruct:\n    sparse = next(bm25.embed(text))\n    return models.PointStruct(\n        id=doc_id,\n        vector={\n            \"dense\": embed_dense(text),\n            \"bm25\": models.SparseVector(\n                indices=sparse.indices.tolist(),\n                values=sparse.values.tolist(),\n            ),\n        },\n        payload=payload,\n    )\n```\n\n`SparseTextEmbedding(\"Qdrant/bm25\")`\n\nis not a neural model. It's a tokenizer plus stemming plus stopword removal, running in a few hundred microseconds per document. The cost of the sparse leg is rounding error next to the dense embedding call.\n\nOne migration note. Moving a few hundred wiki articles and roughly twice as many session memories into the new schema meant re-embedding everything, and re-embedding is exactly where payloads get quietly dropped. My rule: read the full point from the old collection, carry the payload dict forward untouched, and diff payload key sets between source and destination when the run finishes. If a key existed on 300 points before and 280 after, you want a failing assertion, not a shrug. This is the same class of problem I wrote about in [Silent Drift](https://guatulabs.dev/posts/silent-drift-why-re-embedding-only-on-count-changes-rots-your-semantic-index/): count-based checks pass while content quietly diverges.\n\nQdrant does the fusion server-side through prefetch, which saves a round trip and keeps the client dumb:\n\n```\nsparse_q = next(bm25.query_embed(user_query))\n\nresults = client.query_points(\n    collection_name=\"wiki_index_v2\",\n    prefetch=[\n        models.Prefetch(query=embed_dense(user_query), using=\"dense\", limit=20),\n        models.Prefetch(\n            query=models.SparseVector(\n                indices=sparse_q.indices.tolist(),\n                values=sparse_q.values.tolist(),\n            ),\n            using=\"bm25\",\n            limit=20,\n        ),\n    ],\n    query=models.FusionQuery(fusion=models.Fusion.RRF),\n    limit=20,\n    with_payload=True,\n).points\n```\n\nUse `bm25.query_embed()`\n\nfor queries, not `bm25.embed()`\n\n. Query embedding skips the term-frequency weighting that only makes sense for documents. Mixing them up produces results that look plausible and rank badly.\n\nThe fusion itself is about six lines, and it's worth seeing them written out even if Qdrant runs it for you:\n\n``` python\ndef rrf(ranked_lists, k=60):\n    scores = {}\n    for lst in ranked_lists:                 # each list is [doc_id, ...] by rank\n        for rank, doc_id in enumerate(lst, start=1):\n            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)\n    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)\n```\n\nNo score normalization. No tunable alpha weighting dense against sparse. Rank position is the only input.\n\n``` python\nfrom fastembed.rerank.cross_encoder import TextCrossEncoder\n\nreranker = TextCrossEncoder(model_name=\"jinaai/jina-reranker-v2-base-multilingual\")\n\ndef rerank(query: str, candidates, top_n: int = 5):\n    docs = [c.payload[\"text\"] for c in candidates]\n    scores = list(reranker.rerank(query, docs))   # batched ONNX inference\n    ranked = sorted(zip(scores, candidates), key=lambda p: p[0], reverse=True)\n    return [c for _, c in ranked[:top_n]]\n```\n\nThat's the whole reranking stage. FastEmbed ships the ONNX export with the classification head intact, downloads it on first use, and runs it through ONNX Runtime on CPU at roughly 38ms per query-document pair. Twenty candidates batched lands under half a second on a few cores, and it needs no GPU, no separate inference server, and no model-serving deployment to keep alive.\n\nCompare that to the GGUF path: same nominal model, wrong head, silently meaningless scores.\n\n| Configuration | Correct doc in top 3 | Median latency |\n|---|---|---|\n| Dense only (1024d) | 5 / 10 | ~45 ms |\n| BM25 only | 6 / 10 | ~12 ms |\n| Dense + BM25, RRF | 7 / 10 | ~55 ms |\n| Dense + BM25, RRF, reranked | 8 / 10 | ~480 ms |\n\nBM25 alone beating dense alone surprised me. It also makes sense in hindsight: over half my eval queries were literal strings copied out of a config file or an error log, which is BM25's home turf and dense retrieval's blind spot.\n\nHere's the shape of a query that used to fail:\n\n```\nquery: \"ndots:5\"\n\ndense-only ranking:\n  1. Kubernetes Service Discovery Patterns          (cos 0.612)\n  2. DNS Failover With Two Upstreams                (cos 0.598)\n  3. CoreDNS Tuning Notes                           (cos 0.591)\n  ...\n  7. Wildcard DNS + ndots:5: The TLS Nightmare      (cos 0.544)\n\nhybrid + rerank:\n  1. Wildcard DNS + ndots:5: The TLS Nightmare      (rerank  6.81)\n  2. CoreDNS Tuning Notes                           (rerank  1.24)\n  3. Kubernetes Service Discovery Patterns          (rerank  0.37)\n```\n\nBM25 pulled the right article into the candidate pool at sparse rank 1. RRF pushed it to fused rank 2. The cross-encoder, which actually reads the query and the document together, put it first with a score nearly six times the runner-up.\n\nThree separate mechanisms are doing distinct jobs, and it's worth being precise about which does what.\n\n**Sparse retrieval indexes symbols, not meaning.** BM25 scores a document on term frequency scaled by inverse document frequency, with length normalization. `ndots`\n\nappears in one article out of several hundred, so its IDF is enormous and any document containing it rockets to the top. An embedding model does the opposite: it maps rare tokens into a region of vector space defined by their context, which is exactly the behavior you want for synonyms and exactly the behavior you don't want for identifiers. Dense and sparse are not competing implementations of retrieval. They index different properties of the same text.\n\n**RRF fuses ranks because scores are incomparable.** Cosine similarity lives in [-1, 1] and clusters hard around 0.5 to 0.7 for a technical corpus. BM25 scores are unbounded and depend on corpus size, document length, and term rarity. Normalizing them onto a shared scale requires assumptions about their distributions that break whenever the corpus changes. Reciprocal rank fusion sidesteps the problem: it throws the scores away and keeps only the ordering, then sums `1/(k + rank)`\n\nacross both lists. The `k=60`\n\nconstant flattens the curve near the top so that rank 1 versus rank 2 isn't a cliff, which means a document ranked 3rd by both retrievers can outrank a document ranked 1st by one and 40th by the other. Consensus wins over one confident vote, and that's the behavior you want when one leg is guessing.\n\n**Cross-encoders can do what bi-encoders structurally cannot.** Your embedding model is a bi-encoder: query and document are encoded independently, never seeing each other, and compared by cosine distance at the end. That independence is what makes vector search fast, because you precompute every document embedding once. It also means the model never gets to ask \"does this specific document answer this specific question.\" A cross-encoder concatenates query and document into one sequence and runs full attention across both, so query tokens attend directly to document tokens. Far more accurate, and far too slow to run against your whole corpus. Which is precisely why the architecture is retrieve-then-rerank: cheap methods cut several hundred documents down to 20, the expensive method orders those 20.\n\nThat layering also explains why over-retrieval depth matters. Reranking cannot recover a document that never entered the candidate pool. If your prefetch limit is 5, the reranker is just reordering five things, and your recall ceiling is whatever RRF handed it. Twenty per leg is where my eval stopped improving; going to 50 added latency and no additional top-3 hits.\n\n**Test the reranker in isolation before trusting it.** Build a fixture of 10 to 20 query-document pairs where you know the ranking by hand, score them, and check the correlation. That test takes an hour and would have saved me the entire GGUF detour. It also catches the subtler failure where a reranker works fine on prose and falls apart on code blocks.\n\n**ONNX over GGUF for anything with a classification head.** GGUF is a format built around generative decoder inference. Cross-encoders, classifiers, and any model whose value lives in a trained head on top of the backbone should go through ONNX Runtime, where the head is exported with the graph. FastEmbed makes that a one-liner, and running it on CPU means the reranker isn't competing with your LLM for VRAM. I don't need an accelerator to serve retrieval, which matters when the GPU is busy doing actual inference.\n\n**Set Modifier.IDF at creation time.** I'd rather see this documented in bold in every hybrid search tutorial. Missing it does not raise an error, it just makes the sparse leg mediocre in a way you'll blame on BM25 rather than on your config.\n\n**Measure with your queries, not a benchmark.** MTEB scores told me nothing useful about whether retrieval would find the article about a specific kernel parameter. A hand-built eval of 10 real queries with known-correct answers told me everything, and it's small enough to rerun in under a minute after any config change. Keyword-in-top-3 is a crude metric and a good one, because it maps directly to what the agent experiences: the right context is in the window or it isn't.\n\n**Retrieval precision is upstream of everything else in an agent stack.** Better memory decay policies, better tool descriptions, better prompts, none of them compensate for handing the model the wrong three documents. I've come to treat the retrieval layer the way I treat storage: unglamorous, load-bearing, and worth over-engineering slightly. It sits underneath the [memory architecture](https://guatulabs.dev/posts/cognitive-memory-for-agents-vector-search-vs-activation-based-recall/) and the [decay policy](https://guatulabs.dev/posts/eviction-without-deletion-running-an-act-r-decay-policy-for-agent-memory/), and it's the layer that determines whether the rest of the [agent architecture](https://guatulabs.dev/posts/multi-agent-ai-systems-architecture-patterns/) has anything worthwhile to reason over. If you're building this kind of pipeline for something that has to work on a schedule rather than on a weekend, [that's the sort of work I do](https://guatulabs.com/services).\n\n**What surprised me:** the reranker mattered less than adding BM25. Fusion alone took the eval from 5/10 to 7/10; the cross-encoder added the eighth. I'd assumed the fancy neural component would carry the improvement, and instead the win came from a 1994-vintage ranking function that runs in twelve milliseconds and has no parameters to train. The old algorithm knows something the new model doesn't, which is that sometimes the user meant the exact characters they typed.", "url": "https://wpnews.pro/news/hybrid-retrieval-v2-qwen-embeddings-bm25-and-rrf-with-a-fastembed-reranker", "canonical_source": "https://dev.to/futhgar/hybrid-retrieval-v2-qwen-embeddings-bm25-and-rrf-with-a-fastembed-reranker-1702", "published_at": "2026-08-13 16:15:48+00:00", "updated_at": "2026-08-13 16:50:59.407852+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Qwen", "BM25", "FastEmbed", "Qdrant", "Ollama", "Hugging Face", "RRF"], "alternates": {"html": "https://wpnews.pro/news/hybrid-retrieval-v2-qwen-embeddings-bm25-and-rrf-with-a-fastembed-reranker", "markdown": "https://wpnews.pro/news/hybrid-retrieval-v2-qwen-embeddings-bm25-and-rrf-with-a-fastembed-reranker.md", "text": "https://wpnews.pro/news/hybrid-retrieval-v2-qwen-embeddings-bm25-and-rrf-with-a-fastembed-reranker.txt", "jsonld": "https://wpnews.pro/news/hybrid-retrieval-v2-qwen-embeddings-bm25-and-rrf-with-a-fastembed-reranker.jsonld"}}