A pricing table chunk in my index contained the exact answer to the query — the product name three times, the tier name twice, the number the user asked for. It never cracked the lexical top 10. What did rank first was a 60-token section heading that mentioned the product once and nothing else.
The embedding arm wasn't the problem. The reranker never saw the chunk because the candidate set was already wrong. The culprit was BM25 length normalization: the b=0.75
default that ships with Lucene, Elasticsearch, OpenSearch, and every rank_bm25
copy-paste. It was designed for TREC news articles, and your RAG chunk corpus violates the assumption it encodes.
b=0.75
, a 900-token chunk with 3 occurrences of a term scores ~30% b
interpolates between "no length correction" (b=0
) and "full correction" (b=1
). The default assumes long documents are long because they're verbose, not because they cover more ground. Chunked RAG corpora break that assumption.avgdl
is a mean over your chunk length distribution. If you chunk tables, code, and prose into one index, that distribution is bimodal and the mean describes nothing.avgdl
and docCount
are dfs_query_then_fetch
.b
controls how much a document's length discounts its term frequency. The Lucene form:
$$\text{score} = \sum_{t \in q} \text{idf}(t) \cdot \frac{tf_t \cdot (k_1 + 1)}{tf_t + k_1 \cdot \left(1 - b + b \cdot \frac{dl}{avgdl}\right)}$$
The length term sits in the denominator next to tf
. That placement matters: it isn't a small additive correction, it's a divisor that scales with how far the document is from average length. At b=0.75
and avgdl=250
, a 900-token chunk gets a normalizer of 1.2 * (0.25 + 0.75 * 3.6) = 3.54
, versus 0.516
for a 60-token chunk. Seven times the penalty, applied to every query term independently.
K1_DEFAULT, B_DEFAULT = 1.2, 0.75
def tf_component(tf, dl, avgdl, k1=K1_DEFAULT, b=B_DEFAULT):
"""The per-term saturation factor. idf is a separate multiplier."""
norm = k1 * (1 - b + b * dl / avgdl)
return tf * (k1 + 1) / (tf + norm)
AVGDL = 250
heading = dict(tf=1, dl=60) # "Enterprise pricing" section header
table = dict(tf=3, dl=900) # the chunk that actually has the answer
for b in (0.75, 0.5, 0.4, 0.3, 0.0):
h = tf_component(**heading, avgdl=AVGDL, b=b)
t = tf_component(**table, avgdl=AVGDL, b=b)
print(f"b={b:<5} heading={h:.3f} table={t:.3f} winner={'table' if t > h else 'heading'}")
b=0.75 heading=1.451 table=1.009 winner=heading
b=0.5 heading=1.261 table=1.146 winner=heading
b=0.4 heading=1.199 table=1.212 winner=table
b=0.3 heading=1.142 table=1.285 winner=table
b=0.0 heading=1.000 table=1.571 winner=table
The crossover is around b≈0.4
. Everything above it says a chunk that mentions the term three times in 900 tokens is less about that term than one that mentions it once in 60. For a news corpus that's a reasonable prior — a long article that says "Fed" once is probably not about the Fed. For a chunked technical corpus it's backwards: the long chunk is long because it's a table with rows, and the short one is short because it's a heading with no content.
Because avgdl
stops being meaningful the moment your chunk lengths are multimodal. BM25 was fit on collections whose lengths are roughly log-normal and unimodal — one hump, a mean that describes the typical document.
A real RAG index has at least three humps: headings and list fragments at 20–80 tokens, prose chunks pinned near your splitter's target of 400–600, and structural blobs (tables, code, API schemas) that blow past the target because the splitter refuses to cut mid-table. The mean lands in a valley between the humps. Every prose chunk gets a mild penalty, every table gets a savage one, and every heading fragment gets a bonus it did nothing to earn.
This also explains a symptom people misdiagnose as "our chunking is bad": lexical recall that gets worse after you enable structure-aware splitting. Keeping tables intact is the right call for generation quality, and it directly lowers those chunks' BM25 scores.
Don't expect hybrid fusion to rescue you either. Mean-pooled dense embeddings dilute the signal over long inputs for the same structural reason — one relevant sentence in 900 tokens moves the centroid less than one relevant sentence in 60. Both arms share the bias, so fusion averages two correlated errors instead of cancelling them.
Not your corpus. In Lucene it's sumTotalTermFreq(field) / docCount(field)
from the current reader — per field, per shard, and computed from segment-level term statistics that still include deleted documents until a merge rewrites them.
Three consequences that bite in production:
query_then_fetch
, which scores locally on each shard with local stats. Two identical chunks on shards with different length profiles get different scores. Use search_type=dfs_query_then_fetch
to gather global statistics first, or keep small indices single-shard. The difference is largest exactly where it hurts — small indices, where per-shard sample sizes are tiny.avgdl
shifts under you until merges complete. Scores move without any document changing.avgdl
, and a multi_match
blends two different length priors.Because Lucene doesn't store the exact length. BM25Similarity.computeNorm
compresses the field's token count into a single byte with a 4-bit mantissa and 4-bit exponent. Small values round-trip exactly; past a few dozen tokens the buckets widen geometrically. By the time you're at chunk-sized documents, neighboring representable lengths are tens of tokens apart.
Two practical implications. First, micro-tuning b
to three decimals is theater — the length signal feeding it is quantized well below that resolution. Second, "trim 20 tokens of boilerplate off every chunk" usually changes nothing, because the trimmed length lands in the same bucket.
One more wrinkle: discount_overlaps
defaults to true
, so tokens with a zero position increment — synonyms, decompounded forms, word_delimiter_graph
output — don't count toward length. If you added a synonym filter to improve recall, you also silently changed the length norm of every document containing a synonym, and therefore its score on all queries.
Start from the length distribution, not from a grid search. Compute the coefficient of variation of your chunk token counts. Under ~0.3, the default is fine. Above ~0.6, avgdl
is fiction and you should either fix the chunks or lower b
.
My default for a mixed technical corpus: b
in 0.3–0.45, k1
around 0.9. Lower k1
saturates term frequency faster, which is what you want when a chunk is a bounded window — the third occurrence of a term inside 500 tokens is weak extra evidence, unlike the third occurrence in a 5,000-word article.
PUT /chunks
{
"settings": {
"index": {
"number_of_shards": 1,
"similarity": {
"chunk_bm25": {
"type": "BM25",
"k1": 0.9,
"b": 0.35,
"discount_overlaps": true
}
}
}
},
"mappings": {
"properties": {
"body": { "type": "text", "similarity": "chunk_bm25" },
"token_count":{ "type": "integer" },
"chunk_kind": { "type": "keyword" }
}
}
}
Index token_count
and chunk_kind
even if you don't score on them — you need them for the diagnostic below and for splitting length bands into separate fields later.
Measure rank against length directly. For each query, record the length percentile of the chunks BM25 returns and compare it to the length percentile of the known-relevant chunk:
from scipy.stats import spearmanr
def length_rank_bias(all_hits):
ranks, lengths = [], []
for hits in all_hits:
for r, (_, tokens) in enumerate(hits):
ranks.append(r)
lengths.append(tokens)
rho, p = spearmanr(ranks, lengths)
return rho, p # rho >> 0 means longer chunks systematically rank worse
A strongly positive rho across queries with varied gold-chunk lengths is your evidence. Then sweep b
over {0.75, 0.5, 0.35, 0.2}
on a fixed query set and watch recall@50 of the lexical arm alone — not end-to-end answer quality, which is too noisy to attribute at this granularity.
Watch the failure mode on the other side. As b → 0
, a 5,000-token appendix that mentions your term once ranks with a title that is exactly your term. If your corpus has genuine long-tail junk documents, b=0
will surface them. The cleaner fix, when you can afford the reindex, is to make chunk lengths uniform enough that avgdl
means something again — cap oversized structural chunks by splitting tables on row boundaries with a repeated header, and merge heading-only fragments into the section that follows them. Then the default b=0.75
is defensible again.
Because BM25 length normalization treats length as evidence of verbosity, and b=0.75
applies that penalty multiplicatively to every matched term. In a chunked corpus, length is evidence of structure instead — a table, a code block, a dense spec section — so the penalty is inverted relative to reality. It compounds with a bimodal length distribution that makes avgdl
meaningless, per-shard statistics that make scores non-comparable, and byte-quantized norms that make fine tuning pointless. Fix it by narrowing your chunk length distribution where you can, dropping b
to roughly 0.3–0.45 and k1
to ~0.9 where you can't, forcing global statistics with dfs_query_then_fetch
, and verifying with a Spearman correlation between chunk length and lexical rank before and after.