{"slug": "i-measured-the-rag-technique-menu-on-46000-chunks-four-things-mattered", "title": "I measured the RAG technique menu on 46,000 chunks. Four things mattered.", "summary": "A developer built a RAG system over 62 ancient-history books (~46,000 chunks) and measured retrieval techniques against a fixed 161-question test set. The production system ships without hybrid search or reranking, as those techniques were found to be redundant or not cost-effective. Key wins came from strong embedding models, contextual chunk notes, and structure-aware chunking, while techniques like HyDE and hybrid search were rejected after measurement.", "body_md": "Search \"advanced RAG techniques\" and you'll get a list of twenty things: hybrid search, reranking, HyDE, query decomposition, contextual retrieval, RAPTOR, GraphRAG, parent-child chunking, multi-vector retrieval, semantic chunking. They're presented as a menu you work through, each one making your system a bit better.\n\nThey don't compose like that. Techniques at the same pipeline stage mostly *compete* — they fix the same failure, so the second one you add finds nothing left to fix. And the only way to know which one your system actually needs is to measure on your own corpus.\n\nI built a RAG system over 62 ancient-history books (~46,000 chunks) and put every retrieval technique I could through the same door: implement, measure against a fixed 161-question test set, keep or reject, write down the number. This article is the retrieval half of that ledger — what shipped, what got rejected, and what the rejections taught me. My production system has **no hybrid search and no reranker**, and that's a result, not a shortcut.\n\nEverything before the LLM writes a word lives in one of four stages:\n\n```\nA. INGEST  →  B. QUERY TRANSFORM  →  C. RETRIEVAL CORE  →  D. POST-RETRIEVAL\n```\n\nSame stage → the techniques compete. Different stages → they compose. That one rule kills most of the listicle's implied ordering.\n\nThe second rule is about money, and it's asymmetric:\n\nSo query-side cleverness has to clear a far higher bar than ingest-side cleverness. A technique that buys +1.7 points for a paid API call on every query is a worse deal than the same +1.7 for one overnight batch job — even though the leaderboard row looks identical.\n\n**One more thing, and it's the part nobody labels.** Evidence comes in three grades, and mixing them is how bad advice spreads:\n\nHere's the whole retrieval menu with verdicts. The rest of the article explains the interesting rows.\n\n| Technique | Stage | Verdict | Evidence |\n|---|---|---|---|\n| Strong embedding model | C |\nShip — +18 recall@5 |\nmeasured here |\n| Contextual chunk notes + heading path | A |\nShip — synthesis +18.2 |\nmeasured here |\n| Structure-aware chunking, canonical locators | A |\nShip — architecture |\nmeasured smaller |\n| Dedup, lost-in-the-middle reordering | D |\nShip — free, one line each |\nnot worth measuring |\n| Metadata-filtered search | C |\nShip — enables source isolation |\nmeasured here |\n| Cross-encoder reranking | D | Kept, then dropped — +1.7, paid per query |\nmeasured here |\n| Multi-query expansion | B |\nOff by default — +2.1 |\nmeasured smaller |\nHybrid BM25 + RRF |\nC |\nRejected — zero, byte-identical |\nmeasured here |\nHyDE |\nB |\nRejected — −9.7 |\nmeasured smaller |\n| Parent-child / small-to-big | A |\nRejected — completeness 3.22 → 2.67 |\nmeasured smaller |\n| Contextual compression | D |\nRejected — prompt caching already won |\nreasoned away |\n| Multi-vector (ColBERT), SPLADE | A |\nRejected — 10–50× storage |\nreasoned away |\n| Semantic chunking, step-back, embed-summaries | A/B |\nRejected — no expected ROI |\nreasoned away |\n\nThe baseline everything is measured against: naive dense retrieval, 500-token chunks, top-5 — **recall@5 = 35.2%**. Two out of three questions never reached the model with the right passage.\n\nAt four books you can eyeball every quirk. At sixty-two you can't, and running one splitter over raw Gutenberg text means page headers, tables of contents and translator footnotes all leak into your chunks and get embedded as if they were content.\n\nThe architecture that scales puts a hard interface in the middle:\n\n```\nraw book ──(per-format parser)──▶ normalized document tree ──(ONE uniform chunker)──▶ chunks\n```\n\nAll per-book weirdness lives in parser adapters. The chunker is a single well-tested function that never learns which book it's processing: split on structural boundaries in priority order (section → paragraph → sentence), pack greedily to ~500 tokens, never merge across a section wall, never cut mid-sentence.\n\n**Chunking at scale is a parsing problem, not a token-count problem.** The size discussions get all the attention and are the least interesting decision in the stage.\n\nTwo things I'd insist on again. Every chunk carries character offsets into the normalized text — which makes test-set gold spans *chunking-invariant*, so I could change chunking strategy without rewriting the test set. And every chunk carries its canonical citation locator (`Caesar, Gallic War 4.25`\n\n), because classical texts have stable reference schemes that survive every edition. That bought professional citations in answers and mechanical test-set authoring, for the cost of teaching the parsers to recognize book/chapter numbering.\n\nA raw slice from the middle of chapter 12 reads \"he then marched north\" — the embedder has no idea who \"he\" is or which war this is. The fix is an ingest-time pass: a cheap local LLM writes 1–2 sentences situating each chunk, and the embedding covers `context_note + heading_path + chunk_text`\n\ninstead of bare text. 46,159 of 46,170 chunks enriched in one local batch pass on a consumer GPU, **zero query-time cost**.\n\nThe headline result was +1.7 [recall@5](mailto:recall@5). Noise. The internals were not:\n\n| category | Δ recall@5 |\n|---|---|\n| synthesis | +18.2 |\n| literal | +8.7 |\n| multi-hop | +2.1 |\n| synonym | +0.0 |\n| contradiction | −5.3 |\n| cross-book | −9.0 |\n\nSynthesis — the worst category in the project — transformed. Cross-book got worse. Overall ranking sharpened well above noise (recall@1 +5.8, MRR +0.060). **A flat headline hiding ±18-point internals is the normal case, not the exception.** If I'd only looked at the aggregate I'd have called this a no-op and moved on.\n\nThe generation-side number was even more misleading. Answer completeness appeared to *drop*, 4.45 → 4.30. It hadn't. On the 113 questions both runs answered, completeness was flat (4.46 → 4.40). What actually happened: contextual retrieval converted **12 previously-refused questions into answered ones**, and those 12 — the retrieval-starved hard ones — scored 3.42, dragging the mean down while every prior answer held. In-scope false refusals fell from 15.6% to 7.4%.\n\nA win wearing the disguise of a regression. Any change that converts refusals into answers will do this to you, and the only defense is to always compute the metric on the set of questions both runs answered.\n\nThe rest of the stage was cheap to reject. **Parent-child retrieval** (embed small chunks, hand the LLM their parent section) regressed completeness 3.22 → 2.67 on my predecessor project — re-testable, but I'd want evidence of context starvation first. **ColBERT-style multi-vector** costs 10–50× vector storage; rejected on the storage budget without measuring, and I'll call that what it is. **Semantic chunking** and **embedding summaries instead of chunks** were skipped on expected ROI — the latter is subsumed by contextual notes, which keep the original text *and* add the context.\n\nEvery technique here costs an extra LLM call before you've even searched.\n\n**HyDE** — have the model write a hypothetical answer and embed *that* instead of the question — scored **−9.7 recall@5** on my predecessor. The mechanism is worth understanding because it generalizes: HyDE *replaces* your query, discarding the discriminative terms the user actually gave you. It's built for a query/document vocabulary mismatch. If your embedder is strong enough not to have that mismatch, you're throwing away signal and paying a second of latency for the privilege.\n\n**Multi-query expansion** (paraphrase the question n ways, union the results) measured +2.1 — real but marginal, and it's an extra call plus n searches on every request. Left in as an off-by-default flag.\n\n**Query decomposition** and **step-back prompting** I skipped as standalone techniques for a different reason: a retrieval loop that can search more than once does both of these adaptively, driven by what it actually found. Don't hand-build a static version of a behavior a loop gives you for free.\n\nThis is the one that mattered most, and it's one line of configuration.\n\nSwapping the default embedder for a strong one (`qwen3-embedding-8b`\n\n, hosted) took recall@5 from **35.2% to 53%**. The hardest-hit category — modern-English questions against Victorian translation prose — gained **+41.7 points**. Nothing else in this article comes close.\n\nHow I picked it is the transferable part: shortlist by *constraints*, decide by *ablation*. The constraints were concrete — can it serve queries on a cheap CPU container or does it have to be an API, what's the license, does it need instruction prefixes, does the context window fit contextual notes. That produced four candidates. The leaderboard never got a vote in the final decision, because no leaderboard contains Victorian translation prose. Your corpus is the only leaderboard that counts.\n\nTwo footguns cost real projects real quality here:\n\nThe standard 2024-era advice is that hybrid search — keyword BM25 fused with vector search — always wins at scale, especially for rare proper nouns. My corpus is *full* of rare proper nouns (Vercingetorix, Pharsalus) in inconsistent Victorian spellings. I had rejected hybrid once already at 950 chunks, and I wrote down a prediction before running it: at 46k chunks this flips to a win.\n\nIt didn't flip. It returned nothing.\n\n| metric | dense | hybrid | Δ |\n|---|---|---|---|\n| recall@1 | 32.5 | 30.7 | −1.8 |\n| recall@5 | 56.7 | 56.3 | −0.4 |\nrecall@50 (pool) |\n82.4 |\n82.4 |\n0.0 |\n| MRR | 0.580 | 0.561 | −0.019 |\n\nNot \"roughly the same\" — **byte-identical pool recall, category by category.** Every answer BM25 could find by exact token match, the 8B embedder already had. And fusion made the top ranks slightly *worse*, because RRF injects keyword-noise chunks that displace well-ranked dense hits.\n\nThe mechanism is the finding: **where a strong dense retriever misses, the answer is distributed, not keyword-findable** — so BM25 can't reach it either. The hybrid-always-wins advice assumes a weak lexical first stage. With a modern 8B embedder that assumption is just false on this corpus.\n\nNotice what made that result readable at all: `recall@50`\n\n, treated as *pool recall*. Recall@5 alone would have shown −0.4 and left me guessing whether BM25 had contributed new candidates that fusion then mis-ranked. A metric designed to separate \"widened the pool\" from \"reordered the pool\" turned an ambiguous wash into a clean rejection. Design your metrics to distinguish mechanisms, not just to score outcomes.\n\nA cross-encoder rescores your top-50 and returns the best 5. It's the most-recommended technique in RAG, and I measured five of them.\n\n| reranker | host | recall@5 vs no-rerank |\n|---|---|---|\n| qwen3-reranker-0.6b | local | −3.0 |\n| bge-reranker-v2-m3 | local | −2.1 |\n| cohere/rerank-v3.5 | API | +0.0 |\n| cohere/rerank-4-pro | API | +1.7 |\n\n**This is the exact inverse of the embedder gate.** There, the component was so weak that anything better was a huge win. Here the embedder is so strong that a 0.6B cross-encoder is *worse than the 8B embedder's own ranking* — it adds noise. Only a state-of-the-art hosted reranker helps at all, which means shipping reranking means shipping a paid per-query dependency, forever.\n\nOne architectural law came out of this, and it's free to obey: **rerank the same text you embedded.** Scoring the bare chunk text while the index holds contextualized text made the reranker fight the retriever and undid the contextual gains outright (47.9% vs 51.6% on my predecessor). Retrieval and rerank must share a representation.\n\nThe interesting part is why I dropped it. The reranker was kept provisionally for a *specific stated reason*: contextual retrieval had cost me 9 points on cross-book questions, and reranking the top-50 was supposed to recover them. It didn't — cross-book landed at 26.0 against a 34.4 floor, and **no reranker recovered it**, though the pool demonstrably held the answers. Cross-book was a candidate-*pool* problem, not an ordering problem, and reranking cannot surface what isn't in the pool.\n\nSo: the reranker helped a little, everywhere except the place it was hired to help. Marginal, paid, per-query, forever, and falsified in its own rationale. When a stronger generator arrived later it came out of the pipeline entirely.\n\n**One honest caveat.** The model swap and the reranker drop happened in the same run, so that drop was never cleanly isolated — I skipped the arm that would have separated them, for cost. It's a receipt gap in an otherwise complete ledger, and I'd rather name it than let the table imply more rigor than it has.\n\nThe rest of the stage: **dedup** and **lost-in-the-middle reordering** are one line each, free, and I shipped them without measuring. **Contextual compression** — LLM-summarize the retrieved chunks before stuffing them — I rejected by reasoning: prompt caching already makes raw chunks cheap, compression adds latency, and a summary can silently delete the exact sentence you were going to cite.\n\nAfter all of it, here's what actually determined retrieval quality on this corpus:\n\nAnd what didn't matter: hybrid search (zero), four of five rerankers (negative or nil), HyDE (−9.7), and every fashionable ingest architecture I skipped. Not because they're bad techniques — because on *this* corpus the binding constraint was somewhere else. That's the pattern underneath all of it: **at any moment exactly one thing is the binding constraint, and every technique aimed anywhere else returns noise.** The embedder was binding, so fixing it paid 18 points. Once it wasn't, contextual retrieval was marginal, reranking was marginal-and-paid, and hybrid was nothing at all.\n\nThe shipped retrieval stack is boring: contextual dense embeddings, a strong embedder, top-k, metadata filters. No hybrid. No reranker. **recall@5 = 56.7%.**\n\nWhich raises the obvious question, and it's the reason there's a second half to this story: the finished system answers **100% of in-scope questions** and refuses 96% of the unanswerable ones, on a retriever that finds the right passage 56.7% of the time in a single shot.\n\nIt manages that because it doesn't do a single shot. Once retrieval was closed, the remaining headroom turned out to be architectural — and that's the next article.\n\n*I build RAG and LLM-evaluation systems, and I'm available for contract work. Everything above is open: the code, the 161-question golden set, and the full append-only eval log with every run record behind these numbers.*\n\n*If your team is trying to make an LLM answer reliably from your own data — or trying to figure out whether the one you built already can be trusted — reach out: levriabov@zohomail.eu*", "url": "https://wpnews.pro/news/i-measured-the-rag-technique-menu-on-46000-chunks-four-things-mattered", "canonical_source": "https://dev.to/lev_riabov_e6f2883d44b3ab/i-measured-the-rag-technique-menu-on-46000-chunks-four-things-mattered-2266", "published_at": "2026-08-02 19:16:41+00:00", "updated_at": "2026-08-02 19:43:54.839952+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "natural-language-processing", "ai-infrastructure"], "entities": ["RAG", "HyDE", "BM25", "ColBERT", "SPLADE", "Gutenberg"], "alternates": {"html": "https://wpnews.pro/news/i-measured-the-rag-technique-menu-on-46000-chunks-four-things-mattered", "markdown": "https://wpnews.pro/news/i-measured-the-rag-technique-menu-on-46000-chunks-four-things-mattered.md", "text": "https://wpnews.pro/news/i-measured-the-rag-technique-menu-on-46000-chunks-four-things-mattered.txt", "jsonld": "https://wpnews.pro/news/i-measured-the-rag-technique-menu-on-46000-chunks-four-things-mattered.jsonld"}}