# What chunk size should you use for RAG? > Source: > Published: 2026-08-17 15:42:22.542981+00:00 [ Chunking RAG (Retrieval-Augmented Generation) ](/blog/chunking-strategies-decide-what-context-ai-sees/) ### Chunking Strategies Decide What Context AI Sees Key takeaway There is no single best chunk size for RAG, and the published studies disagree by more than most defaults admit. A 25-configuration chemistry study found recursive 100-token chunks with zero overlap won outright. A 36-approach study across six domains found paragraph-group splitting reached nDCG@5 of 0.459 against under 0.244 for fixed-size character splitting. A NAACL 2025 paper found semantic chunking's cost unjustified, with fixed-size often better on real documents. The reconciliation is that token-aware fixed splitting is a strong baseline, character-blind splitting is not, and the winning configuration is corpus-dependent. The larger finding is that restoring context a boundary strips out beats tuning where the boundary falls: Anthropic measured a 67% reduction in retrieval failures that way, far more than any size change reports. Every RAG tutorial gives you a chunk size. Most give you 512 tokens with 10 to 20 percent overlap, stated as though it were settled. It is not settled. The published studies disagree with each other, and with the tutorials, by margins large enough to change which documents your system can answer questions about. That matters because [chunking](/glossary/chunking/) happens once, at ingestion. Unlike a prompt you can revise or a reranker you can swap, a chunk size decision is baked into the index until you re-embed the whole corpus. It is worth getting deliberately rather than by copying a default. Three recent results point in different directions, and the disagreement is the most useful thing about them. [Chunk Twice, Embed Once](https://arxiv.org/abs/2506.17277) (Amiri and Bocklitz, 2025) swept 25 chunking configurations across five method families for chemistry-aware retrieval. The winner was recursive token-based chunking at **100 tokens with no overlap**, which outperformed the alternatives while carrying minimal resource overhead. That is one fifth the size and none of the overlap that most guides recommend. [A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity](https://arxiv.org/abs/2603.06976) (Shaukat, Adnan and Kuhn, 2026) tested 36 segmentation approaches across six knowledge domains and five embedding models. Paragraph group chunking reached mean **nDCG@5 around 0.459 with Precision@1 near 24%**, while a fixed-size character baseline came in **below 0.244 nDCG@5 with 2 to 3% Precision@1**. On that evidence, how you split matters enormously and fixed sizes look indefensible. [Is Semantic Chunking Worth the Computational Cost?](https://arxiv.org/abs/2410.13070) (Qu, Tu and Bao, NAACL 2025 Findings) evaluated semantic chunking across document retrieval, evidence retrieval and retrieval-based answer generation. It concluded the computational cost is not justified by consistent gains. The improvements clustered on synthetic datasets stitched for high topic diversity, and **fixed-size chunking often performed better on documents that resemble real ones**. So: fixed size wins, fixed size loses badly, and the sophisticated alternative is not worth paying for. All three are credible. The contradiction dissolves under two distinctions the summaries usually flatten. Most of the apparent disagreement comes from two different methods sharing one label. The baseline that lost badly in the 2026 study was fixed-size *character* chunking, which cuts blind to word and sentence boundaries. The baseline that held up in the NAACL paper was fixed-size splitting that respects separators. Those are not the same technique, and comparing a result about one to a result about the other produces a contradiction that does not exist. Recursive splitting, which walks a hierarchy of separators like paragraph then sentence then word before falling back to a token count, is the strong version, and it is the one worth treating as your baseline. A winning configuration does not transfer between corpora, which is why a chemistry result should not set your default. Chemistry literature is dense, technical and structurally consistent, exactly the shape where small chunks retrieve precisely because each one is a self-contained claim. A corpus of loosely structured meeting notes behaves differently: the same 100-token cut lands mid-discussion and strips the referent. The 2026 study spread across six knowledge domains precisely because the authors expected domain to matter, and it did. Chunk size interacts with the embedding model, so the two are one decision rather than two. The same 2026 investigation swept its 36 segmentation approaches against five different embedding models, and the effective ranking of strategies was not stable across them. A size that performs well under one model’s tokenizer and context length can underperform under another, which means inheriting a chunk size from a tutorial written against a different model is inheriting an untested assumption. What survives both distinctions is a narrower claim: **use recursive, separator-aware splitting, and treat the size as a parameter you measure rather than a number you inherit.** Recursive splitting at 256 to 512 tokens is the defensible default for a mixed corpus, not because it is optimal but because it is rarely badly wrong. The bands below describe how retrieval behavior changes as size moves, drawn from the ranges the studies above actually tested. Treat them as a place to start measuring rather than a recommendation to adopt. | Chunk size | Behavior | When it fits | |---|---|---| | Under 128 tokens | Precise retrieval, high fragmentation risk | Dense technical text, factual lookups | | 256 to 512 tokens | Strong general default | Most mixed corpora | | 512 to 1024 tokens | Diluted embeddings, fewer results per window | Long-form reasoning over prose | | Over 1024 tokens | Matches broadly and imprecisely | Rarely the right answer | The reason the middle band is a reasonable default is not that it is optimal. It is that it is rarely badly wrong, which is a different and more useful property when you have not measured yet. Overlap may not be earning its cost, and the best-performing configuration in a 25-way sweep used **zero**. The standard advice is 10 to 20 percent, offered to stop an idea being severed by a boundary, but that framing counts only the benefit. The cost side is rarely stated. Duplicated content inflates index size roughly in proportion to the overlap fraction, so 20 percent overlap is a 20 percent larger index to store, embed and search. That much is obvious. The less obvious cost is what it does to your results: near-identical passages compete for the same top-k slots, so a query can return three variants of one paragraph instead of three distinct pieces of evidence. For a single-fact lookup that is merely wasteful. For a question that needs to combine facts from different parts of a corpus, it is actively harmful, because the redundancy crowds out the diversity the question depends on. There is a second-order effect worth naming. Overlap is usually introduced to compensate for a boundary that damages meaning, which means it is treating a symptom. If the boundary is placed on structure the author already chose, headings and paragraphs and code blocks, the ideas are far less likely to be severed in the first place and there is proportionally less for overlap to rescue. Teams that fix boundary placement often find they can reduce overlap without losing anything. Overlap is a parameter to test, not a default to inherit. Start at zero, measure, and add it only if your retrieval metrics ask for it. Restoring the context a boundary removes produces far larger measured gains than moving the boundary does. Anthropic’s [contextual retrieval](https://www.anthropic.com/engineering/contextual-retrieval) work reported a top-20 retrieval failure rate of **5.7%** for a standard chunking and embedding pipeline, then cut it to **1.9%** with two changes that leave the split point untouched. That is a **67% reduction**, and no chunk-size result in this literature reports an effect close to it. The intermediate step matters for understanding why. Prepending a short chunk-specific description before embedding, so a passage about a raised limit carries which limit and which product it refers to, took the failure rate from 5.7% to **3.7%** on its own, a 35% reduction. Adding contextual BM25 alongside it and then a reranker took it the rest of the way to 1.9%. Each layer restores something the boundary had discarded: the first restores the referent, the second restores lexical matching the embedding blurred, the third restores ordering the retriever got wrong. The practical consequence is an ordering. If you have limited time, fix what the boundary strips out before you tune where the boundary falls. A team that sweeps chunk sizes from 128 to 1024 and picks the winner is optimizing within a few points. A team that adds chunk-specific context is playing for a 35% reduction in retrieval failures before touching size at all. Late chunking attacks the same problem from the embedding side, running the whole document through a long-context embedding model before splitting the resulting token embeddings, so each chunk embedding carries information from its neighbors and stays useful even when the boundary itself is imprecise. Both techniques share a premise worth stating plainly: the problem was never the size of the chunk, it was that the chunk stopped making sense on its own. Measure on your own corpus, in this order, because the sequence matters more than any individual setting. The steps below move from cheapest to most expensive, and each one is worth exhausting before moving to the next. Chunk size looks like a configuration value and behaves like an architectural decision. It determines the smallest unit of meaning your system can retrieve, which sets a ceiling on every answer that follows. Tuning it at the splitter is treating a [context engineering](/glossary/context-engineering) problem as a parameter problem. The alternative is to make the retrieved unit self-sufficient in the first place, so that a boundary landing in an awkward place costs less. Splitting a document into entries that keep their relationships means an agent can call `wire_navigate` to walk from a match into related entries rather than depending on one chunk having happened to contain both halves of the answer, which is the failure overlap and size tuning are both trying to paper over ([how Wire’s retrieval is measured](/why-wire/retrieval-benchmarks/)). That does not make chunk size irrelevant. It makes it a smaller decision, which given how much the literature disagrees about the right answer is the more comfortable place to be. Sources: [Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation (arXiv 2506.17277)](https://arxiv.org/abs/2506.17277) · [A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity (arXiv 2603.06976)](https://arxiv.org/abs/2603.06976) · [Is Semantic Chunking Worth the Computational Cost? (arXiv 2410.13070, NAACL 2025 Findings)](https://arxiv.org/abs/2410.13070) · [Anthropic: Contextual Retrieval](https://www.anthropic.com/engineering/contextual-retrieval) Related Claude Code, Codex & Cursor wire-memory writes decisions, corrections, and preferences to a container as you work, keyed to you and your project. Any agent you connect can look them up when it needs to.