Chunking: Getting the First Cut Right

A developer's guide to RAG chunking argues that the document-splitting step is the most commonly rushed and most damaging part of a retrieval pipeline, since errors there propagate downstream into embeddings and retrieval. It compares three approaches — fixed-size, recursive, and semantic chunking — with code samples for each, noting that semantic chunking uses embedding cosine distances and a percentile threshold to cut where topics shift rather than at arbitrary character counts.

Let's start with an inconvenient truth: your carefully built RAG system, the one you're quietly proud of, might be sabotaging itself at step one. Not at retrieval. Not at generation. At the very first thing that happens to your document, the cut . Chunking sounds boring. It's the "take a big document, slice it into smaller pieces" step, and it's usually the part we rush through to get to the "fun" stuff like embeddings and vector databases. But here's the thing nobody tells loudly enough: get this step wrong, and everything downstream is trying to fix a problem that got created on page one. A 40-page PDF is a lot to ask an embedding model to capture in a single vector. Embedding models have limits, and even without those limits, cramming a whole document into one vector means everything specific gets averaged into mush. So documents get broken down into smaller, digestible segments called chunks, which then get embedded and stored so a retrieval system can search and pull back only the relevant piece instead of the whole haystack. In short: chunking exists because "smaller and specific" beats "big and vague" when you're trying to find a needle. The argument isn't about whether to chunk, rather it's about how . If you've read literally any RAG tutorial, you've met these three: Fixed-size chunking : the "just cut every N characters" approach. Fast, simple, and about as thoughtful as slicing a birthday cake with your eyes closed. If a sentence happens to cross the 512-token limit, well... it doesn't get much of a say in where the cut lands. js import { CharacterTextSplitter } from "@langchain/textsplitters"; const splitter = new CharacterTextSplitter { separator: "", // no separator to respect, just slice chunkSize: 512, chunkOverlap: 50, } ; const documents = await splitter.createDocuments documentText ; Recursive chunking : a slightly more polite cousin. It tries to split along natural boundaries first like paragraphs, then sentences, then lines and only falling back to a hard cut if it absolutely has to. Think of it as tearing a loaf of bread along the lines where it naturally wants to break, instead of taking a knife to it wherever you feel like. js import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters"; const splitter = new RecursiveCharacterTextSplitter { chunkSize: 100, chunkOverlap: 20, separators: "\n\n", "\n", " ", "" , // tried in this order } ; const documents = await splitter.createDocuments documentText ; Semantic chunking : the fancy one. Instead of counting characters, it uses a model to notice where the topic shifts, and cuts there instead, so each chunk stays thematically whole rather than just structurally intact. python import OpenAI from "openai"; const openai = new OpenAI ; function cosineDistance a: number , b: number : number { const dot = a.reduce sum, v, i = sum + v b i , 0 ; const magA = Math.sqrt a.reduce sum, v = sum + v v, 0 ; const magB = Math.sqrt b.reduce sum, v = sum + v v, 0 ; return 1 - dot / magA magB ; } function percentile values: number , p: number : number { const sorted = ...values .sort a, b = a - b ; const idx = Math.floor p / 100 sorted.length - 1 ; return sorted idx ; } async function semanticChunk sentences: string , percentileThreshold = 95 : Promise<string { const { data } = await openai.embeddings.create { model: "text-embedding-3-small", input: sentences, } ; const embeddings = data.map d = d.embedding ; const distances = embeddings .slice 0, -1 .map emb, i = cosineDistance emb, embeddings i + 1 ; const threshold = percentile distances, percentileThreshold ; const breakpoints = distances .map d, i = d threshold ? i : -1 .filter i = i == -1 ; const chunks: string = ; let start = 0; for const bp of breakpoints { chunks.push sentences.slice start, bp + 1 .join " " ; start = bp + 1; } chunks.push sentences.slice start .join " " ; return chunks; } These three are the bread and butter of chunking tutorials, and they're not wrong, they're just not the whole story anymore. Because here's the problem none of them actually solve. Say you've got a wiki-style document about the Pain Assault Arc in Naruto Shippuden . Somewhere deep in that document, chunked out on its own, lives this sentence: "He entered Sage Mode and used it to counter the attack, saving the village." Cool. Except... who's "he"? Which attack? Which village? The names "Naruto" and "Pain" and "Konoha" all got mentioned two paragraphs earlier, in a completely different chunk. This isolated little sentence has no idea it's talking about Naruto's fight against Pain. Its embedding doesn't know either, and here's the part that actually matters, neither does a keyword index, because the words "Naruto" and "Pain" never appear in this chunk's text at all. Hybrid search doesn't help here, because there's nothing to match against for either retrieval method This isn't a rare edge case. It's the default behavior of naive chunking, and it's arguably the single biggest reason RAG systems either return nothing useful or hallucinate with total confidence. Fixed-size, recursive, and semantic chunking all have this same blind spot, they decide where to cut, but none of them fix what the chunk loses in the process . Two real fixes for this showed up in 2024, and both are super cool. Anthropic published this idea in September 2024, and the core insight is almost too simple: before you embed a chunk, have a fast LLM read the whole document and write a brief context, usually 50 to 100 tokens, explaining what this specific chunk is about and where it sits in the bigger picture. Something like “This section discusses X in the context of document Y about Z.” Then you add the context to the chunk before embedding it. So instead of embedding: You embed: "This chunk is from the Pain Assault Arc from Naruto Shippuden,set in Konoha, describing Naruto's fight against Pain. He entered Sage Mode and used it to counter the attack, saving the village." Same underlying fact, but a very different embedding. The first one could look like a million other generic "hero powers up and saves the day" sentences floating around in vector space, it could belong to almost any shonen protagonist, in almost any arc. The second one is much more clearly "Naruto, Pain Assault Arc, Sage Mode, Konoha" because now the context is actually part of the text. The numbers here are genuinely worth your attention. Anthropic's own testing found that adding these contextual embeddings alone cut the retrieval failure rate meaningfully; stacking contextual embeddings with a contextual version of keyword search BM25 can reduce the failure rate by 49%; and adding a reranking step on top of both took it down even further by 67%, bringing what started as roughly a 1-in-17 miss rate down to closer to 1-in-50. That's a pretty big jump for something that, at first glance, sounds like "just add a little context to the chunk." And it doesn't seem to be only a benchmark thing, either. Some engineering teams that tried the technique on their own datasets have reported solid double-digit improvements too. So, thankfully, this isn't one of those papers where the chart looks beautiful and your production system politely refuses to cooperate. Here's what that looks like Reranked Contextual Embedding and Contextual BM25 reduces the top-20-chunk retrieval failure rate by 67%. Source: Anthropic https://www.anthropic.com/engineering/contextual-retrieval There is a catch, though: Contextual Retrieval isn't free. You're making an LLM call for each chunk, and at large scale, especially with documents that change frequently, that cost can start to matter. At the same time, prompt caching changes the math quite a bit. If you're processing a document's chunks together, the source document can stay cached while you generate context for each chunk, making those follow-up calls much cheaper. So the cost is often closer to a one-time indexing expense than something you pay every time someone asks a question. Worth testing on your own documents before assuming the published numbers transfer directly. Here's a genuinely clever reversal, published by Jina AI around the same time in 2024. Instead of chunking first and embedding each piece separately the way literally everyone does it , late chunking flips the order: embed the entire document first using a long-context embedding model, and only split it into chunks after , right before the final pooling step that produces the vector. Here's how it works: Source: arxiv.org https://arxiv.org/pdf/2409.04701 Why does this matter? Because when you embed the whole document at once, the model's internal representation of any given sentence already "knows" about the sentences around it, the pronouns, the company names, the earlier context...before chunking ever happens. Splitting afterward means each resulting chunk embedding still carries traces of that full-document understanding, without needing a separate LLM call to explain it. Same problem as Contextual Retrieval, solved from the opposite direction, no extra generation step, just a smarter place to make the cut. The tradeoff: this needs an embedding model that can actually handle long context in the first place thousands of tokens, not the traditional 512-token ceiling , so it's less of a drop-in fix and more of a "does your embedding model support this" decision. Honest answer: it depends on what you're optimizing for. And the classic fixed/recursive/semantic split from earlier? Still relevant, they're about where you cut. Contextual Retrieval and Late Chunking are about what the chunk remembers after the cut . You genuinely need both decisions, and mixing them say, recursive chunking plus contextual embeddings is completely normal. Chunking isn't the boring setup step before the "real" RAG work starts. It's the step that decides whether everything downstream like retrieval, reranking, generation is working with chunks that actually know what they're talking about, or chunks that are quietly missing context about their own document. Fix it here, and you'll notice the rest of the pipeline getting easier to trust.