{"slug": "i-turned-both-knobs-on-my-on-prem-chinese-rag-all-the-way-up-it-didn-t-get-a", "title": "I turned both knobs on my on-prem Chinese RAG all the way up. It didn't get a single extra question right", "summary": "An engineer evaluating an on-premises Chinese retrieval-augmented generation (RAG) system for a client found that increasing retrieval depth and upgrading the reranker did not improve accuracy. A ten-second probe revealed that the correct answer was often ranked highly by raw vector similarity, and deeper candidate sets did not help because the reranker failed to rank the correct passage in the top five. The engineer concluded that candidate depth is necessary but not sufficient, and that reranking costs scale linearly with candidate count, leading to significant latency increases without accuracy gains.", "body_md": "If you're about to \"increase the retrieval depth\" or \"swap in a newer reranker\" to improve your Chinese RAG accuracy, **spend ten seconds on this first**:\n\nTake one question you got wrong, and compute **where the passage containing the correct answer ranks in your raw vector similarity**.\n\nNo reranker. No full pipeline. One embedding call and one round of cosine similarity.\n\nThat number decides everything downstream:\n\nWithout that probe I'd have swept `top-k = 10 / 20 / 30 / 50 / 80 / 100`\n\n, six configurations, 29 questions each, tens of seconds per question — **about an hour**. After the ten-second probe, those six configurations collapsed to **one**.\n\nI was evaluating a \"the data never leaves the building\" knowledge base for a client, and wrote up my research as a handoff doc for the next session. The to-do was specific:\n\nStep 1: top-k 10 → 50 (zero cost, zero VRAM)— a public benchmark on 3,493 traditional-Chinese questions shows a weak embedding plus a reranker reaching 98%+ at k=100, beating every single-stage retriever.\n\n⚠️ Don't just crank it to the max (a SIGIR 2025 paper reports that over-reranking degrades results); there's a sweet spot and you have to sweep for it.\n\nStep 2:(+1.3GB, same architecture, same API, no prompt changes)`bge-reranker-base`\n\n→`bge-reranker-v2-m3`\n\nIt reads as completely reasonable. External benchmarks behind it, a counter-example warning, a clear execution order.\n\n**Both steps ended up as \"don't.\"** And that \"zero cost\" in step 1 is wrong.\n\nBefore touching anything I looked at the table from my previous round:\n\n| Configuration | K=1 | K=3 | K=5 |\n|---|---|---|---|\n| Vector only | 84% | 88% | 92% |\nVector + rerank (candidate depth 30) |\n88% |\n96% |\n96% |\n| Vector + rerank (candidate depth 10) | 84% | 92% | 92% |\n\n96% is 24 out of 25. **And the miss is always the same question.**\n\nThat sentence matters, because it **caps** the question \"how much can deeper candidates buy me?\" — **at most one question, i.e. 4 percentage points.**\n\nSo the question changed from \"which k is best?\" into something far cheaper: **did the reranker ever even see the correct answer for that one question?**\n\nI wrote a ten-second probe: for each of the 25 questions, where does the passage containing the correct answer rank by raw vector similarity?\n\n| Rank of the first correct passage | Questions |\n|---|---|\n| 1st | 22 |\n| 3rd / 5th / 21st | 1 each |\n83rd |\n1 (the usual suspect) |\n\nOne table cut six configurations down to one:\n\nAnd check the calibration: depth 10's ceiling is 92%, measured 92%. Depth 30's ceiling is 96%, measured 96%. **Both match to the decimal.**\n\nAt that moment I derived a conclusion: *\"whatever's in the candidate set, the reranker pulls into the top 5 ⇒ this pipeline's bottleneck is how deep we fish, not how good the reranker is.\"*\n\nThe derivation itself is sound (the miss set must contain the out-of-candidates set, and the two have equal size, so they're the same set).\n\n**Then I ran depth 100, and it falsified that conclusion.**\n\n| Depth 30 | Depth 100 | |\n|---|---|---|\n| Retrieval hit rate (K=5) | 96% (24/25) | 96% (24/25) |\n| Miss | that question | that question |\n| Median latency per question | 6,834 ms | 37,651 ms |\n\nThe correct answer ranks 83rd, so it **did** make it into the 100 candidates. The reranker saw it. **And then ranked it outside the top 5.**\n\nSo the right statement isn't the one I derived — it's a more useful one:\n\nCandidate depth is necessary, not sufficient.\n\nFor that question to land, two things must both hold: ① you fish deep enough to see it, and ② the reranker is willing to rank it near the top.\n\nDepth 30 fails at ①. Depth 100 clears ① and fails at ②.\n\nIncidentally: **\"zero cost\" is wrong.** A reranker is a cross-encoder, and its cost is **linear in the candidate count** — send it 100 candidates and it computes \"how relevant is this passage to this question\" 100 times. Going from depth 30 to 100 took each question from 6.8 to 37.7 seconds: **5.5× the latency, for zero extra correct answers.**\n\n\"Zero VRAM\" is right. \"Zero cost\" is not. And what it spends is precisely the one budget in this setup that had already bottomed out.\n\nIf the bottleneck is the reranker, replace it. My to-do said `bge-reranker-v2-m3`\n\n.\n\n**First discovery: that model isn't available on my stack.** I run ONNX through fastembed on CPU, and its cross-encoder list has six models — no v2-m3. Only two are multilingual: the `bge-reranker-base`\n\nI'm already on (a 2023 base-tier model), and `jinaai/jina-reranker-v2-base-multilingual`\n\n(1.11 GB, **2024 generation, marketed on multilingual**).\n\nActually using v2-m3 would mean swapping the whole service from ONNX to PyTorch. That's a separate project. So I swapped in the jina model — same API, one parameter changed, no prompt edits.\n\n**It lost in three directions at once:**\n\n| reranker | K=1 |\nK=3 | K=5 | Pure Chinese prose questions, K=1 | Latency/question |\n|---|---|---|---|---|---|\n`bge-reranker-base` (2023) |\n88% |\n96% | 96% |\n88.9% (8/9) |\n6,834 ms |\n`jina-reranker-v2-multilingual` (2024) |\n76% |\n96% | 96% |\n66.7% (6/9) |\n8,219 ms |\n\n**But the fourth reason is the fatal one, and I nearly missed it.**\n\nMy previous round had turned up a bonus finding: **the reranker's score itself works as a \"refuse to answer without calling the LLM\" gate.** Below a threshold, return \"not found\" — skip generation entirely. Fast, and structurally incapable of making things up.\n\nFor this client that matters more than usual, because making things up is exactly what they're afraid of.\n\nThe scoring script prints a \"median top-1 score\" line each round:\n\n`bge-reranker-base`\n\n: answerable 5.26 / unanswerable −2.14`jina`\n\n: answerable 0.84 / unanswerable −0.76That looks like \"the gap got smaller.\" What I wrote in the report at the time was \"this doesn't let us say jina made the gate worse [unknown].\"\n\n**But the statistic that finding needs isn't the median.** It needs **the minimum score among answerable questions** against **the maximum among unanswerable ones** — because two distributions can have very different medians and still have overlapping tails.\n\nThe per-question scores were already in the scoring run's JSON. I computed it afterwards at zero extra cost:\n\n| reranker | Answerable min\n|\nUnanswerable max\n|\nGap | Can one threshold separate them? |\n|---|---|---|---|---|\n`bge-reranker-base` |\n−0.5237 | −1.8008 | +1.2771 |\n✅ yes\n|\n`jina-v2-multilingual` |\n−0.9005 | −0.6736 | −0.2269 |\n❌ overlap\n|\n| Vector only (no rerank) | 0.5895 | 0.6377 | −0.0482 | ❌ overlap |\n\n**Switching to jina means: K=1 down 12 points, latency up 1.2×, and then you also forfeit the entire refusal defence.**\n\nAnd the median made it look like it merely \"got smaller.\" That's the most valuable lesson of this round: **the statistic a report prints is not necessarily the statistic your conclusion needs.** I turned it into a command (reads the stored scoring JSON, zero cost) and left a test inside it: fixtures of answerable `[9, 9, 9, −1]`\n\nand unanswerable `[0]`\n\n— medians differ by 9, looks trivially separable; the truth is min −1 < max 0, **overlap**. That test exists so that if anyone ever switches it back to medians, it goes red.\n\nI had one hypothesis left. Looking at the top-5 for that question across four configurations, I noticed **all five slots in every configuration were filled by the same class of file** — my own blog drafts. Derivative articles discussing the same incidents, not the canonical records of those incidents.\n\nThat matches a published finding: **what hurts RAG isn't \"irrelevant,\" it's \"relevant but wrong.\"** It also matches my own earlier observation that this corpus writes about the same event a dozen times over.\n\nSo I added an exclusion rule, filtered the entire blog-source directory out of the candidates, and re-ran that question.\n\n**Still missed.** And the correct answer didn't even make the filtered 30 candidates — excluding the derivative material just pushed **other non-answers** up.\n\nThree roads walked:\n\n| Hypothesis | Measured | Conclusion |\n|---|---|---|\n| Not deep enough | depth 100 got it into candidates, reranker still didn't rank it | ❌ necessary, not sufficient |\n| Reranker too old | swapped to a 2024 multilingual model; still missed, every other metric regressed | ❌ actively worse |\n| Corpus noise drowning the canon | excluded the biggest source of derivatives; it can't even reach candidates | ❌ hypothesis falsified\n|\nEmbedding not good enough |\nuntested | ⬅️ only one left\n|\n\nThat question asks \"what has to be done to a temp table containing personal data for it to count as safe, and which incident forced that?\" — and the file with the answer is anchored on a database table name. **The question doesn't contain that table name; only the answer does.** It's a pure semantic-bridging task, and my embedding model puts it 83rd.\n\n**This round proved the embedding is the bottleneck, by elimination** — every knob outside the embedding was turned to the stop with no effect. My original plan wrote the precondition for \"swap the embedding\" as \"only swap once you've proven embedding is the bottleneck.\" That precondition now holds; it just got proven by exclusion.\n\nMy next cut will be **query rewriting** (have the LLM expand the question into a hypothetical answer first, then embed that) rather than swapping the embedding outright — much cheaper, and among the open-source projects I've read, the one that handles Chinese most seriously does its work on the **query construction** side (part-of-speech weighting, positional proximity weighting), not by changing models.\n\nThis round also turned up something I wasn't looking for.\n\nTo get a same-day control, I re-ran the depth-30 configuration. Recall was identical (96%, same question missed) — **comparability confirmed**.\n\nBut **latency was 6,834 ms per question. A week earlier I'd measured the same configuration at 18,152 ms.**\n\n**Same model, same corpus, same question bank, same laptop. A factor of 2.66.**\n\nTwo consequences:\n\nI don't know the source of the variance yet (thermal throttling? something else running? a package version?). **All I can state is that corpus, question bank, and model are ruled out**, because all three were re-verified identical the same day.\n\nSo the rule now is: **any latency number going into a quote must have its own control, measured in the same round, on the same machine, with no other load.**\n\nFour scoring runs, three probes, roughly an hour of machine time. What it bought:\n\n**Don't touch either knob.** Stay on `bge-reranker-base`\n\nwith candidate depth 30.\n\nThat sounds like nothing happened. In practice I got three things:\n\nPlus one observation I can sell: **derivative copies inside a client's corpus** (drafts, meeting notes, several versions of the same policy) will systematically outrank the canon — my own blog drafts pushing my own canonical records out of the top 5 is the live example. But **don't sell \"curate the corpus\" as a cure-all**, because in the same round I measured it doing nothing for that question.\n\nIf you're doing this kind of measure-before-you-change work, I've written others in the same family: [my test report printed \"0/96, 0% pass rate\" — the truth was my account was out of credit](https://dev.to/content/zero-of-ninetysix-was-an-empty-wallet-en), on why a red light needs as much suspicion as a green one; and [the A/B test where I cut 41 tool descriptions in half](https://dev.to/content/slimming-skill-descriptions-ab-test-en), which is the same discipline applied to a change I was confident about — an absolute number can't tell you whether something dropped, so you have to go build the control arm.", "url": "https://wpnews.pro/news/i-turned-both-knobs-on-my-on-prem-chinese-rag-all-the-way-up-it-didn-t-get-a", "canonical_source": "https://dev.to/dexterlung/i-turned-both-knobs-on-my-on-prem-chinese-rag-all-the-way-up-it-didnt-get-a-single-extra-question-18im", "published_at": "2026-09-01 13:05:15+00:00", "updated_at": "2026-09-01 13:24:41.006221+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-infrastructure", "developer-tools"], "entities": ["bge-reranker-base", "bge-reranker-v2-m3", "SIGIR"], "alternates": {"html": "https://wpnews.pro/news/i-turned-both-knobs-on-my-on-prem-chinese-rag-all-the-way-up-it-didn-t-get-a", "markdown": "https://wpnews.pro/news/i-turned-both-knobs-on-my-on-prem-chinese-rag-all-the-way-up-it-didn-t-get-a.md", "text": "https://wpnews.pro/news/i-turned-both-knobs-on-my-on-prem-chinese-rag-all-the-way-up-it-didn-t-get-a.txt", "jsonld": "https://wpnews.pro/news/i-turned-both-knobs-on-my-on-prem-chinese-rag-all-the-way-up-it-didn-t-get-a.jsonld"}}