This model’s maximum context length is 8192 tokens, however you requested 8451 tokens
. You changed a model string, not a chunker, and now the ingest job is failing on documents it processed last week. The chunker was never measuring what you thought it was measuring; the old model was just forgiving enough to hide it.
The loud failure is an over-length rejection at the embedding or completion endpoint. The exact wording differs by provider and has changed within providers, so match on the status and the error type rather than on the sentence, but the shape is always the same: a declared maximum, a requested count, and a difference.
The quiet failure is worse and far more common. Some endpoints do not reject an over-length input; they truncate it and return a normal result. Nothing errors. Your chunk went in at 9,000 tokens, 8,192 of it was embedded, and the last ninth of the text — which may have been the part with the answer in it — is in no vector anywhere. Retrieval quality drops by an amount nobody can attribute, months later, with no log line to point at.
Whether a given endpoint truncates or rejects is a per-provider, per-endpoint behaviour and it is not stable across versions. Do not infer it from one successful call. Test it deliberately with a deliberately over-length input and record what happened, because the audit below depends on knowing which one you are dealing with.
Chunkers acquire wrong budgets in a small number of recognisable ways.
size
, but the retrieval step that concatenates k
of them pays for the overlap k
times.This is a search problem before it is a maths problem, because the assumption is rarely in one place. Sweep for all of these:
rg -n '\b(256|384|512|768|1000|1024|1536|2000|2048|4000|4096|8000|8191|8192|16384|32000|32768|100000|128000|200000)\b' \
--glob '!**/node_modules/**' --glob '!**/*.lock'
rg -n '(/\s*4|\* *0\.25|CHARS_PER_TOKEN|approx.*token|estimate.*token)'
rg -n '(get_encoding|cl100k|o200k|p50k|r50k|AutoTokenizer\.from_pretrained)'
rg -n '(chunk_size|chunkSize|max_tokens|maxTokens|max_completion_tokens|top_k|topK|overlap)'
For each hit, write down three things: what the number is supposed to bound, which model or endpoint it was derived for, and who enforces it at runtime. Most audits find the same number written in four places — the chunker, the retriever, the prompt builder and a test fixture — and only one of them was updated.
Replace the constant with an equation that names every term. For a retrieval-augmented completion:
W = model context window, in tokens
R = tokens reserved for the completion (your output cap)
S = system prompt, measured
T = tool schema block, measured (often the forgotten term)
H = conversation history you intend to keep
k = number of chunks retrieved
o = overlap tokens per chunk
M = safety margin
usable_for_chunks = W - R - S - T - H - M
chunk_size = usable_for_chunks / k
effective_cost_of_k_chunks = k * chunk_size - (k - 1) * o # best case, adjacent
= k * chunk_size # worst case, scattered
Two terms are routinely omitted. T
, the serialised tool definitions, is invisible in the prompt you wrote but real in the request body, and it grows every time someone adds a tool — its size is worth measuring rather than guessing, as the token cost of a tool schema sets out. M
matters because you are counting with a tokenizer and the provider is counting with theirs; even when they agree on the vocabulary they may not agree on the exact overhead per message. Reserve a few percent rather than aiming at the cap.
Two edge cases fall out of the equation and are worth handling explicitly. Documents shorter than chunk_size
produce a single chunk and no overlap, so a corpus of short records never exercises the overlap path at all — which is why the bug ships. And a chunker that splits on structure rather than length can emit one oversized chunk from a single unbreakable element: a wide table, a base64 blob, a minified asset. Give the splitter a hard fallback that splits mid-element when no boundary exists inside the budget, and log every time it fires, because a frequently-firing fallback means the boundary rules are wrong for that document type.
For the embedding side, the equation is simpler and the term that bites is different: there is no completion to reserve for, so the budget is the model’s input cap, but the correct target is usually well below it for reasons that have nothing to do with fitting — see migrating chunk size when you change embedding models.