{"slug": "chunking-getting-the-first-cut-right", "title": "Chunking: Getting the First Cut Right", "summary": "A developer's guide to RAG chunking argues that the document-splitting step is the most commonly rushed and most damaging part of a retrieval pipeline, since errors there propagate downstream into embeddings and retrieval. It compares three approaches — fixed-size, recursive, and semantic chunking — with code samples for each, noting that semantic chunking uses embedding cosine distances and a percentile threshold to cut where topics shift rather than at arbitrary character counts.", "body_md": "Let's start with an inconvenient truth: your carefully built  RAG system, the one you're quietly proud of, might be sabotaging itself at step one. Not at retrieval. Not at generation. At the very first thing that happens to your document, *the cut*.\n\nChunking sounds boring. It's the \"take a big document, slice it into smaller pieces\" step, and it's usually the part we rush through to get to the \"fun\" stuff like embeddings and vector databases. But here's the thing nobody tells loudly enough: *get this step wrong, and everything downstream is trying to fix a problem that got created on page one.*\n\nA 40-page PDF is a lot to ask an embedding model to capture in a single vector. Embedding models have limits, and even without those limits, cramming a whole document into one vector means everything specific gets averaged into mush. So documents get broken down into smaller, digestible segments called chunks, which then get embedded and stored so a retrieval system can search and pull back only the relevant piece instead of the whole haystack.\n\nIn short: chunking exists because \"smaller and specific\" beats \"big and vague\" when you're trying to find a needle. The argument isn't about *whether* to chunk, rather it's about *how*.\n\nIf you've read literally any RAG tutorial, you've met these three:\n\n**Fixed-size chunking :** the \"just cut every N characters\" approach. Fast, simple, and about as thoughtful as slicing a birthday cake with your eyes closed. If a sentence happens to cross the 512-token limit, well... it doesn't get much of a say in where the cut lands.\n\n``` js\nimport { CharacterTextSplitter } from \"@langchain/textsplitters\";\n\nconst splitter = new CharacterTextSplitter({\n  separator: \"\", // no separator to respect, just slice\n  chunkSize: 512,\n  chunkOverlap: 50,\n});\n\nconst documents = await splitter.createDocuments([documentText]);\n```\n\n**Recursive chunking :** a slightly more polite cousin. It tries to split along natural boundaries first like paragraphs, then sentences, then lines and only falling back to a hard cut if it absolutely has to. Think of it as tearing a loaf of bread along the lines where it naturally wants to break, instead of taking a knife to it wherever you feel like.\n\n``` js\nimport { RecursiveCharacterTextSplitter } from \"@langchain/textsplitters\";\n\nconst splitter = new RecursiveCharacterTextSplitter({\n  chunkSize: 100,\n  chunkOverlap: 20,\n  separators: [\"\\n\\n\", \"\\n\", \" \", \"\"], // tried in this order\n});\n\nconst documents = await splitter.createDocuments([documentText]);\n```\n\n**Semantic chunking :** the fancy one. Instead of counting characters, it uses a model to notice where the *topic* shifts, and cuts there instead, so each chunk stays thematically whole rather than just structurally intact.\n\n``` python\nimport OpenAI from \"openai\";\n\nconst openai = new OpenAI();\n\nfunction cosineDistance(a: number[], b: number[]): number {\n  const dot = a.reduce((sum, v, i) => sum + v * b[i], 0);\n  const magA = Math.sqrt(a.reduce((sum, v) => sum + v * v, 0));\n  const magB = Math.sqrt(b.reduce((sum, v) => sum + v * v, 0));\n  return 1 - dot / (magA * magB);\n}\n\nfunction percentile(values: number[], p: number): number {\n  const sorted = [...values].sort((a, b) => a - b);\n  const idx = Math.floor((p / 100) * (sorted.length - 1));\n  return sorted[idx];\n}\n\nasync function semanticChunk(sentences: string[], percentileThreshold = 95): Promise<string[]> {\n  const { data } = await openai.embeddings.create({\n    model: \"text-embedding-3-small\",\n    input: sentences,\n  });\n  const embeddings = data.map((d) => d.embedding);\n\n  const distances = embeddings\n    .slice(0, -1)\n    .map((emb, i) => cosineDistance(emb, embeddings[i + 1]));\n\n  const threshold = percentile(distances, percentileThreshold);\n  const breakpoints = distances\n    .map((d, i) => (d > threshold ? i : -1))\n    .filter((i) => i !== -1);\n\n  const chunks: string[] = [];\n  let start = 0;\n  for (const bp of breakpoints) {\n    chunks.push(sentences.slice(start, bp + 1).join(\" \"));\n    start = bp + 1;\n  }\n  chunks.push(sentences.slice(start).join(\" \"));\n  return chunks;\n}\n```\n\nThese three are the bread and butter of chunking tutorials, and they're not wrong, they're just not the whole story anymore. Because here's the problem none of them actually solve.\n\nSay you've got a wiki-style document about *the Pain Assault Arc in Naruto Shippuden*. Somewhere deep in that document, chunked out on its own, lives this sentence: \"He entered Sage Mode and used it to counter the attack, saving the village.\"\n\nCool. Except... who's \"he\"? Which attack? Which village? The names \"Naruto\" and \"Pain\" and \"Konoha\" all got mentioned two paragraphs earlier, in a completely different chunk. This isolated little sentence has no idea it's talking about Naruto's fight against Pain. Its embedding doesn't know either, and here's the part that actually matters, neither does a keyword index, because the words \"Naruto\" and \"Pain\" never appear in this chunk's text at all. Hybrid search doesn't help here, because there's nothing to match against for either retrieval method\n\nThis isn't a rare edge case. It's the default behavior of naive chunking, and it's arguably the single biggest reason RAG systems either return nothing useful or hallucinate with total confidence. Fixed-size, recursive, and semantic chunking all have this same blind spot, *they decide where to cut, but none of them fix what the chunk loses in the process*.\n\nTwo real fixes for this showed up in 2024, and both are super cool.\n\nAnthropic published this idea in September 2024, and the core insight is almost too simple: before you embed a chunk, have a fast LLM read the *whole* document and write a brief context, usually 50 to 100 tokens, explaining what this specific chunk is about and where it sits in the bigger picture. Something like “This section discusses X in the context of document Y about Z.”  Then you add the context to the chunk before embedding it.\n\nSo instead of embedding:\n\nYou embed:\n\n\"This chunk is from the Pain Assault Arc from Naruto Shippuden,set in Konoha, describing Naruto's fight against Pain. He entered Sage Mode and used it to counter the attack, saving the village.\"\n\nSame underlying fact, but a very different embedding. The first one could look like a million other generic \"hero powers up and saves the day\" sentences floating around in vector space, it could belong to almost any shonen protagonist, in almost any arc. The second one is much more clearly \"Naruto, Pain Assault Arc, Sage Mode, Konoha\" because now the context is actually part of the text.\n\nThe numbers here are genuinely worth your attention. Anthropic's own testing found that adding these contextual embeddings alone cut the retrieval failure rate meaningfully; stacking contextual embeddings *with* a contextual version of keyword search (BM25) can reduce the failure rate by 49%; and adding a reranking step on top of both took it down even further by 67%, bringing what started as roughly a 1-in-17 miss rate down to closer to 1-in-50. \n\nThat's a pretty big jump for something that, at first glance, sounds like \"just add a little context to the chunk.\"\n\nAnd it doesn't seem to be only a benchmark thing, either. Some engineering teams that tried the technique on their own datasets have reported solid double-digit improvements too. So, thankfully, this isn't one of those papers where the chart looks beautiful and your production system politely refuses to cooperate.\n\nHere's what that looks like\n\nReranked Contextual Embedding and Contextual BM25 reduces the top-20-chunk retrieval failure rate by 67%.\n\n**Source:** [Anthropic](https://www.anthropic.com/engineering/contextual-retrieval)\n\nThere is a catch, though: Contextual Retrieval isn't free. You're making an LLM call for each chunk, and at large scale, especially with documents that change frequently, that cost can start to matter. At the same time, prompt caching changes the math quite a bit. If you're processing a document's chunks together, the source document can stay cached while you generate context for each chunk, making those follow-up calls much cheaper.\n\nSo the cost is often closer to a one-time indexing expense than something you pay every time someone asks a question. Worth testing on your own documents before assuming the published numbers transfer directly.\n\nHere's a genuinely clever reversal, published by Jina AI around the same time in 2024. Instead of chunking first and embedding each piece separately (the way literally everyone does it), late chunking flips the order: embed the *entire* document first using a long-context embedding model, and only split it into chunks *after*, right before the final pooling step that produces the vector.\n\nHere's how it works:\n\n**Source:** [arxiv.org](https://arxiv.org/pdf/2409.04701)\n\nWhy does this matter? Because when you embed the whole document at once, the model's internal representation of any given sentence already \"knows\" about the sentences around it, the pronouns, the company names, the earlier context...before chunking ever happens. Splitting afterward means each resulting chunk embedding still carries traces of that full-document understanding, without needing a separate LLM call to explain it. Same problem as Contextual Retrieval, solved from the opposite direction, no extra generation step, just a smarter place to make the cut.\n\nThe tradeoff: this needs an embedding model that can actually handle long context in the first place (thousands of tokens, not the traditional 512-token ceiling), so it's less of a drop-in fix and more of a \"does your embedding model support this\" decision.\n\nHonest answer: it depends on what you're optimizing for.\n\nAnd the classic fixed/recursive/semantic split from earlier? Still relevant, they're about *where* you cut. Contextual Retrieval and Late Chunking are about *what the chunk remembers after the cut*. You genuinely need both decisions, and mixing them (say, recursive chunking *plus* contextual embeddings) is completely normal.\n\nChunking isn't the boring setup step before the \"real\" RAG work starts. It's the step that decides whether everything downstream like retrieval, reranking, generation is working with chunks that actually know what they're talking about, or chunks that are quietly missing context about their own document. Fix it here, and you'll notice the rest of the pipeline getting easier to trust.", "url": "https://wpnews.pro/news/chunking-getting-the-first-cut-right", "canonical_source": "https://dev.to/sekharendu_dey/chunking-getting-the-first-cut-right-pbc", "published_at": "2026-09-24 14:30:48+00:00", "updated_at": "2026-09-24 14:59:27.686662+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "natural-language-processing"], "entities": ["LangChain", "OpenAI", "CharacterTextSplitter", "RecursiveCharacterTextSplitter", "text-embedding-3-small"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/chunking-getting-the-first-cut-right", "markdown": "https://wpnews.pro/news/chunking-getting-the-first-cut-right.md", "text": "https://wpnews.pro/news/chunking-getting-the-first-cut-right.txt", "jsonld": "https://wpnews.pro/news/chunking-getting-the-first-cut-right.jsonld"}}