# Chunking: the most underrated decision in your RAG pipeline > Source: > Published: 2026-08-24 06:53:12+00:00 # Chunking: the most underrated decision in your RAG pipeline Ask a team how their RAG pipeline works and they will tell you about the embedding model, the vector database, and maybe the reranker. Ask them how they chunk their documents and you will usually get "uh, 500 tokens with some overlap? Whatever the default was." That default is quietly deciding the q Ask a team how their RAG pipeline works and they will tell you about the embedding model, the vector database, and maybe the reranker. Ask them how they chunk their documents and you will usually get "uh, 500 tokens with some overlap? Whatever the default was." That default is quietly deciding the quality of every answer the system gives. Chunking is the highest-leverage, least-discussed decision in a RAG pipeline, and I want to convince you of that with concrete examples rather than hand-waving. Say your docs contain this refund policy: ## Refund policy Customers may return items within 30 days of delivery for a full refund. Items must be unopened and in original packaging. Opened electronics are subject to a 15% restocking fee. Sale items are final and cannot be returned unless defective. Defective items can be returned within 90 days regardless of sale status. Now run it through a fixed-size chunker, the kind that cuts every N characters. Depending on where the boundary lands, you can get a chunk like this: original packaging. Opened electronics are subject to a 15% restocking fee. Sale items are final and cannot be returned unless A user asks "can I return a sale item?" The retriever finds this chunk (it literally contains "Sale items are final and cannot be returned unless") and hands it to the model. The model reads it and answers "sale items are final and cannot be returned." The critical exception, "unless defective," was decapitated by a character boundary. The 90-day defective window lives in a different chunk that scored lower and never made it into the prompt. Nothing in your stack is broken. The embedding model is fine, the vector database is fine, the LLM did exactly what the context told it to. The answer is still wrong, and it is wrong because of an off-by-one in a splitting function nobody has looked at since the prototype. A heading-aware chunker would have kept the whole "Refund policy" section together as one chunk, and the model would have seen both the rule and the exception. Same stack, same models, correct answer. Your instinct might be "fine, make chunks bigger so nothing gets cut." That trades one failure mode for a sneakier one. Here is the mental model: an embedding compresses a chunk into a single point in vector space. That point represents the average meaning of the chunk. So: Small chunks give you precise retrieval but amnesiac context. A two-sentence chunk about restocking fees embeds into a sharp, specific point; a question about restocking fees lands right next to it. But once retrieved, it may not carry enough surrounding context for the model to answer. "It must be unopened" is useless when "it" was defined a paragraph earlier. Large chunks give you context but diluted similarity. A 2,000-token chunk covering refunds, shipping, and warranties embeds into a mushy midpoint of all three topics. A precise question about restocking fees is now near-ish that blob but not close to anything, and an unrelated but tighter chunk can outrank it. You also burn prompt budget: retrieve three 2,000-token chunks and you have shipped 6,000 tokens to the model to answer a question about one sentence. There is no universally correct size. There is only a tradeoff you should be making deliberately: for most prose documentation, somewhere between 200 and 500 tokens is a sane starting point, then you measure (more on that at the end). The standard mitigation for boundary cuts is overlap: each chunk repeats the last 10 to 20 percent of the previous one, so a sentence straddling a boundary appears whole in at least one chunk. Overlap helps, and you should probably use some. But notice what it costs: Storage and money. 15 percent overlap means embedding and storing 15 percent more tokens, forever, on every re-ingestion. Near-duplicate retrieval. Two overlapping chunks are semantically almost identical, so they retrieve together. Your top 3 can effectively become the top 2, with one slot wasted on a copy. It does not fix the real problem. Overlap patches arbitrary cuts; it does not make cuts less arbitrary. It is a bandage on a splitter that does not understand the document. Which brings us to the actual fix. Documents are not streams of characters. They have structure: headings, sections, paragraphs, list items, table rows. The counterintuitive part is that the best chunking algorithm is barely an algorithm; it is respect for structure the author already gave you. Structure-aware chunking means: split on headings first. If a section is too long, split on paragraphs. If a paragraph is somehow still too long, split on sentences. Only cut at a character count as an absolute last resort. Every mainstream framework has a version of this (recursive character splitting with separators ordered from most to least meaningful), and markdown-header splitters do it natively for docs. For HTML or markdown documentation, heading-aware chunking alone eliminates the entire class of mid-sentence, mid-thought failures from the refund example. Sections are the units authors used to organize meaning; chunks that match them inherit that coherence for free. Structure gives you one more gift. Once you split by headings, you know each chunk's position in the document hierarchy, and you can prepend it as a small header before embedding: type Section = { pageTitle: string; path: string[]; body: string }; function withContextHeader(s: Section): string { // "Returns & Refunds > Refund policy > Sale items" const breadcrumb = [s.pageTitle, ...s.path].join(" > "); return `${breadcrumb}\n\n${s.body}`; } // Embed the contextualized text, but keep the raw body too const chunks = sections.map((s) => ({ text: withContextHeader(s), // what gets embedded and shown to the LLM raw: s.body, // handy for display/citations source: s.pageTitle, })); Consider a chunk whose body is just "Yes, within 30 days, in original packaging." Embedded alone, that text is meaningless; it could be about returning shoes or renting scaffolding. Embedded as "Returns & Refunds > Refund policy > Sale items" plus the body, it now lives near every returns-related query in vector space, and when it lands in the prompt the model knows what "yes" refers to. This trick costs a few dozen tokens per chunk and routinely rescues short, context-dependent sections (FAQ answers are the classic case, since half of them start with "Yes" or "No"). Anthropic's "contextual retrieval" work is a fancier version of the same idea, using an LLM to write a chunk-specific context sentence, but the humble breadcrumb gets you a surprising share of the benefit for free. Here is where most teams stop: they eyeball three answers, feel good, and ship. Then they argue about chunk size in Slack for a year, with vibes as the only evidence. Chunking is measurable, and the measurement is not even hard. You need a golden question set: 30 to 50 real questions, each labeled with the document (or section) that contains the answer. Then you measure retrieval hit rate: for each question, did any of the top-k retrieved chunks come from the labeled source? type GoldenQuestion = { question: string; expectedSource: string }; async function hitRate(golden: GoldenQuestion[], k = 5): Promise { let hits = 0; for (const g of golden) { const results = await retrieve(g.question, k); // your retrieval fn if (results.some((r) => r.source === g.expectedSource)) hits++; } return hits / golden.length; } Now chunking changes become experiments instead of opinions. Re-chunk with a different strategy, re-embed, run the golden set, compare one number. Fixed 500-character chunks score 62 percent, heading-aware chunks score 78 percent, heading-aware plus contextual headers scores 84 percent (numbers like these are typical of what you will see on your own corpus, and the deltas are the point, not the absolute values). Two practical notes. First, source your golden questions from real user queries if you have any, because real users phrase things worse than you do, and that is exactly what retrieval must survive. Second, keep the eval fast and run it on every ingestion change, the same way you run unit tests on every commit. A retrieval eval that requires a notebook and an afternoon will be run twice and then never again. If your RAG answers are mediocre, the reflex is to reach for a better embedding model or a better LLM. Check your chunks first. Pull ten of them at random out of your index and read them. If a chunk would confuse you without extra context, it is confusing the embedding model twice as much. The playbook, in order of effort: split on structure instead of character counts; keep chunks in the few-hundred-token range for prose; add modest overlap only where structure is missing; prepend breadcrumb headers before embedding; and build the golden-set eval so every future change is a measurement, not a debate. Models get all the attention because they are the exciting part. Chunking is the unglamorous part that determines what the model gets to read, and no model can answer from a paragraph that was cut in half. I work on Fetchply, an AI support agent for ecommerce, where heading-aware chunks with breadcrumb headers beat every clever alternative we have tested. ## Key Takeaways - •Ask a team how their RAG pipeline works and they will tell you about the embedding model, the vector database, and maybe the reranker - •This story was reported by **Dev.to**, covering developments in the** dev**space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: [Read Full Article on Dev.to →](https://dev.to/fetchply/chunking-the-most-underrated-decision-in-your-rag-pipeline-1eg7)