Chunking: the most underrated decision in your RAG pipeline Chunking is the highest-leverage, least-discussed decision in a RAG pipeline, according to a technical analysis, because fixed-size chunk boundaries can cut critical exceptions from retrieved context, leading to incorrect answers even with a well-functioning embedding model, vector database, and LLM. The article argues that heading-aware chunking preserves entire sections, while chunk size involves a tradeoff between precise retrieval and contextual completeness, recommending 200 to 500 tokens for most prose documentation and the use of overlap to mitigate boundary cuts. Chunking: the most underrated decision in your RAG pipeline Ask a team how their RAG pipeline works and they will tell you about the embedding model, the vector database, and maybe the reranker. Ask them how they chunk their documents and you will usually get "uh, 500 tokens with some overlap? Whatever the default was." That default is quietly deciding the q Ask a team how their RAG pipeline works and they will tell you about the embedding model, the vector database, and maybe the reranker. Ask them how they chunk their documents and you will usually get "uh, 500 tokens with some overlap? Whatever the default was." That default is quietly deciding the quality of every answer the system gives. Chunking is the highest-leverage, least-discussed decision in a RAG pipeline, and I want to convince you of that with concrete examples rather than hand-waving. Say your docs contain this refund policy: Refund policy Customers may return items within 30 days of delivery for a full refund. Items must be unopened and in original packaging. Opened electronics are subject to a 15% restocking fee. Sale items are final and cannot be returned unless defective. Defective items can be returned within 90 days regardless of sale status. Now run it through a fixed-size chunker, the kind that cuts every N characters. Depending on where the boundary lands, you can get a chunk like this: original packaging. Opened electronics are subject to a 15% restocking fee. Sale items are final and cannot be returned unless A user asks "can I return a sale item?" The retriever finds this chunk it literally contains "Sale items are final and cannot be returned unless" and hands it to the model. The model reads it and answers "sale items are final and cannot be returned." The critical exception, "unless defective," was decapitated by a character boundary. The 90-day defective window lives in a different chunk that scored lower and never made it into the prompt. Nothing in your stack is broken. The embedding model is fine, the vector database is fine, the LLM did exactly what the context told it to. The answer is still wrong, and it is wrong because of an off-by-one in a splitting function nobody has looked at since the prototype. A heading-aware chunker would have kept the whole "Refund policy" section together as one chunk, and the model would have seen both the rule and the exception. Same stack, same models, correct answer. Your instinct might be "fine, make chunks bigger so nothing gets cut." That trades one failure mode for a sneakier one. Here is the mental model: an embedding compresses a chunk into a single point in vector space. That point represents the average meaning of the chunk. So: Small chunks give you precise retrieval but amnesiac context. A two-sentence chunk about restocking fees embeds into a sharp, specific point; a question about restocking fees lands right next to it. But once retrieved, it may not carry enough surrounding context for the model to answer. "It must be unopened" is useless when "it" was defined a paragraph earlier. Large chunks give you context but diluted similarity. A 2,000-token chunk covering refunds, shipping, and warranties embeds into a mushy midpoint of all three topics. A precise question about restocking fees is now near-ish that blob but not close to anything, and an unrelated but tighter chunk can outrank it. You also burn prompt budget: retrieve three 2,000-token chunks and you have shipped 6,000 tokens to the model to answer a question about one sentence. There is no universally correct size. There is only a tradeoff you should be making deliberately: for most prose documentation, somewhere between 200 and 500 tokens is a sane starting point, then you measure more on that at the end . The standard mitigation for boundary cuts is overlap: each chunk repeats the last 10 to 20 percent of the previous one, so a sentence straddling a boundary appears whole in at least one chunk. Overlap helps, and you should probably use some. But notice what it costs: Storage and money. 15 percent overlap means embedding and storing 15 percent more tokens, forever, on every re-ingestion. Near-duplicate retrieval. Two overlapping chunks are semantically almost identical, so they retrieve together. Your top 3 can effectively become the top 2, with one slot wasted on a copy. It does not fix the real problem. Overlap patches arbitrary cuts; it does not make cuts less arbitrary. It is a bandage on a splitter that does not understand the document. Which brings us to the actual fix. Documents are not streams of characters. They have structure: headings, sections, paragraphs, list items, table rows. The counterintuitive part is that the best chunking algorithm is barely an algorithm; it is respect for structure the author already gave you. Structure-aware chunking means: split on headings first. If a section is too long, split on paragraphs. If a paragraph is somehow still too long, split on sentences. Only cut at a character count as an absolute last resort. Every mainstream framework has a version of this recursive character splitting with separators ordered from most to least meaningful , and markdown-header splitters do it natively for docs. For HTML or markdown documentation, heading-aware chunking alone eliminates the entire class of mid-sentence, mid-thought failures from the refund example. Sections are the units authors used to organize meaning; chunks that match them inherit that coherence for free. Structure gives you one more gift. Once you split by headings, you know each chunk's position in the document hierarchy, and you can prepend it as a small header before embedding: type Section = { pageTitle: string; path: string ; body: string }; function withContextHeader s: Section : string { // "Returns & Refunds Refund policy Sale items" const breadcrumb = s.pageTitle, ...s.path .join " " ; return ${breadcrumb}\n\n${s.body} ; } // Embed the contextualized text, but keep the raw body too const chunks = sections.map s = { text: withContextHeader s , // what gets embedded and shown to the LLM raw: s.body, // handy for display/citations source: s.pageTitle, } ; Consider a chunk whose body is just "Yes, within 30 days, in original packaging." Embedded alone, that text is meaningless; it could be about returning shoes or renting scaffolding. Embedded as "Returns & Refunds Refund policy Sale items" plus the body, it now lives near every returns-related query in vector space, and when it lands in the prompt the model knows what "yes" refers to. This trick costs a few dozen tokens per chunk and routinely rescues short, context-dependent sections FAQ answers are the classic case, since half of them start with "Yes" or "No" . Anthropic's "contextual retrieval" work is a fancier version of the same idea, using an LLM to write a chunk-specific context sentence, but the humble breadcrumb gets you a surprising share of the benefit for free. Here is where most teams stop: they eyeball three answers, feel good, and ship. Then they argue about chunk size in Slack for a year, with vibes as the only evidence. Chunking is measurable, and the measurement is not even hard. You need a golden question set: 30 to 50 real questions, each labeled with the document or section that contains the answer. Then you measure retrieval hit rate: for each question, did any of the top-k retrieved chunks come from the labeled source? type GoldenQuestion = { question: string; expectedSource: string }; async function hitRate golden: GoldenQuestion , k = 5 : Promise