In this article, you will learn how to choose the right chunking strategy for a retrieval-augmented generation system by working through a structured decision tree tied to the properties of your documents and retrieval task.
Topics we will cover include:
- What each major chunking strategy assumes about your documents, and when those assumptions hold.
- A six-question decision tree that maps concrete document and retrieval conditions to specific chunking strategies.
- The failure signals that indicate a chosen strategy is not working, and the fixes that tend to resolve them.
Introduction #
When you are building a retrieval-augmented generation system, you often work with documents that are larger than what an embedding model or LLM context window can accommodate, so you need to split them into smaller chunks. Chunk size and overlap may seem like straightforward configuration choices, but the way you set those boundaries directly affects retrieval. Poor boundaries can cause retrieval to miss the passage that answers a query, or return a chunk that is technically relevant but lacks the surrounding context needed to make it useful.
A chunking strategy defines how you decide where to split a document, and that choice can have a major effect on retrieval quality. When the boundaries preserve the right context, embeddings, vector search, and generation have a stronger basis to work with. When the boundaries break important context, reranking and prompt tuning cannot fully recover what was lost during splitting. No single chunking method works well for every document. The right choice, therefore, depends on the structure of the document and the retrieval task.
This article covers:
- What each major chunking strategy assumes about your documents
- A decision tree that maps every one of those strategies to a concrete condition worth checking
- How the strategies land on real document types, with tradeoffs laid out side by side
- The failure signals that tell you a chosen strategy isn’t working
We start with why this choice is important.
What Each Chunking Strategy Assumes #
Each chunking strategy is based on a specific assumption about the document. Choosing a strategy that matches the document’s structure and needs helps improve retrieval quality while avoiding unnecessary complexity:
| Strategy | What It Assumes |
|---|---|
| Fixed-size chunking | Boundary placement doesn’t matter much, since overlap can patch whatever gets cut mid-thought |
| Recursive chunking | Paragraph and sentence boundaries are a safe enough proxy for meaning when no stronger structure is available |
| Document-based chunking | The document’s own structure — headers, sections, function boundaries — lines up with its semantic structure |
| Semantic chunking | Topic shifts are detectable sentence by sentence through embedding distance |
| Late chunking | A chunk’s meaning depends on the whole document, not just its own text |
| Adaptive chunking | A single chunk size can’t serve sections that vary sharply in information density |
| Hierarchical chunking | Retrieval needs to answer both broad and specific questions from the same corpus |
| LLM-based chunking | Only a model reading the text can find logical breakpoints that structure and embeddings miss |
| Agentic chunking | The collection is varied enough that no single rule handles all of it well |
When the strategy fits the document, its added complexity can be worthwhile. When it does not, it adds overhead without improving retrieval. For example, semantic chunking is unnecessary for a short FAQ entry, while fixed-size chunking can split related clauses in a contract. These mismatches can lead to retrieval problems that originate in preprocessing rather than generation.
Working Through a Decision Tree to Choose a Chunking Strategy #
The tree has six branching questions, each narrowing the strategy space based on a concrete property of the document or the retrieval task. Every question below ends with a direct answer: which strategy to use if the condition holds, and where to go next if it doesn’t. Following them in order helps you choose a strategy based on the document and retrieval task rather than defaulting to a familiar option.
Question 1: Does the Document Need Chunking at All?
Short, self-contained content — such as FAQ entries, product descriptions, or single config snippets — often fits comfortably inside one embedding call and covers exactly one topic.
If the document sits well within the model’s context window and doesn’t mix subjects, embed it whole and skip chunking. Splitting content that is already coherent only adds retrieval overhead. If the document is too long, or covers more than one topic, move to Question 2.
Question 2: Does the Document Have a Clear Internal Structure?
Some documents already contain natural boundaries that reflect how their content is organized. Markdown headings, HTML elements, PDF sections and tables, and code functions or classes can all provide meaningful chunk boundaries.
If the document has a reliable structure that maps to logical content units, use document-based chunking. The document is parsed according to its format, keeping sections, paragraphs, tables, functions, or other structural elements together. If the structure is inconsistent, incomplete, or doesn’t provide useful boundaries, move to Question 3.
Question 3: Do Chunks Need Context From Elsewhere in the Document?
Some documents carry dependencies that span the whole text. Take, as an example, a legal contract that defines a term in section one and uses it in section eight, or a technical spec whose terminology only makes sense next to an earlier paragraph.
If retrieval needs to preserve that document-wide relationship, use late chunking. Embed the full document with a long-context model first, then derive chunk-level embeddings from that result, so each chunk still carries a trace of the whole document. If chunks can stand on their own without losing meaning, move to Question 4.
Question 4: Does Information Density Vary, or Do You Need Multiple Levels of Granularity?
This question covers two different conditions, each pointing to a different strategy:
- If information density varies across sections, use adaptive chunking to dynamically adjust parameters such as chunk size and overlap to match the content. Dense, information-rich sections can use smaller, more granular chunks, while broader or less dense sections can use larger chunks.
- If the same corpus needs to support both broad questions (such as what a document covers) and specific questions (such as exact syntax), use hierarchical chunking , with coarse parent chunks and finer-grained child chunks.
If neither condition applies, move to Question 5.
Question 5: Does Retrieval Quality Matter More Than Cost and Latency?
This is the cost and quality tradeoff. If retrieval quality is the priority and the budget supports a model call per document, use LLM-based chunking to identify logical breakpoints or atomic propositions. If quality still matters but per-document model calls are too costly, and topic shifts are unpredictable, use semantic chunking based on sentence-embedding distance. It is less costly than an LLM pass while still accounting for meaning. If speed and cost are the priority, move to Question 6.
Question 6: Is This One Document Type, or a Heterogeneous Collection?
If the corpus mixes document types where the right strategy differs from one document to the next, use [agentic chunking](https://www.ibm.com/think/topics/agentic-chunking). For example, a collection might contain structured reports, narrative transcripts, and code. An agent can inspect each document and select the appropriate approach.
If the collection is reasonably uniform, [recursive chunking](https://weaviate.io/blog/chunking-strategies-for-rag#recursive-chunking) is a practical default. It splits content at paragraph, sentence, and then word boundaries until the chunks fit the target size, while preserving natural structure without requiring per-document analysis.
If speed matters more than chunk quality, [fixed-size chunking](https://weaviate.io/blog/chunking-strategies-for-rag#fixed-size-chunking-or-token-chunking) is a reasonable fallback for quick prototypes. After working through these questions, the decision tree should look like so:
Mapping the Decision Tree to a Chunking Strategy #
Working through the tree lands you on one of the strategies below. Systems often blend more than one, but the table maps each strategy to the situation it fits best and the cost that comes with it:
| Strategy | Best For | Key Tradeoff |
|---|---|---|
| No chunking | Short, single-topic content that already fits the context window | None — skips preprocessing entirely |
| Document-based chunking | Content with reliable structural markup (Markdown, HTML, code) | Breaks down when headers don’t track topic boundaries |
| Late chunking | Documents with cross-references or dependencies spanning sections | Requires a long-context embedding model |
| Adaptive chunking | Corpora with sharply uneven information density | Added indexing and retrieval complexity |
| Hierarchical chunking | Corpora needing both broad and specific queries answered well | Two levels of chunks to build and keep in sync |
| LLM-based chunking | Complex, high-stakes documents where retrieval quality justifies the cost | Highest compute cost per document |
| Semantic chunking | Documents with unpredictable topic shifts on a moderate budget | Needs threshold tuning to avoid inconsistent chunk counts |
| Agentic chunking | Heterogeneous collections where the optimal strategy varies by document | Requires per-document analysis before chunking begins |
| Recursive chunking | General-purpose default for a reasonably uniform corpus | Less semantically precise than meaning-aware methods |
| Fixed-size chunking | Fast prototyping when speed matters more than chunk quality | Ignores structure and meaning entirely |
The right chunking strategy depends on the structure, complexity, and retrieval needs of your data, with the goal of balancing retrieval quality, computational cost, and implementation complexity.
Recognizing Common Chunking Pitfalls #
A strategy that looked right on paper can still misbehave once it meets actual documents. The table below lists the signals worth watching for, what they usually mean, and the fix that tends to work:
| Symptom | Likely Cause | Fix |
|---|---|---|
| Chunks consistently cut mid-sentence or mid-table | A structure-blind method applied to a structured document | Switch to recursive chunking instead of widening overlap |
| Semantic chunking produces wildly inconsistent chunk counts across similar documents | Similarity threshold is miscalibrated | Tighten the threshold, or fall back to LLM-based boundaries for the documents that keep misbehaving |
| Document-based chunks range from one line to several pages | Headers don’t reliably track topic boundaries | Combine document-based splitting with a recursive pass inside oversized sections |
| Hierarchical or adaptive chunking adds latency without improving answers | Corpus is small and fairly uniform | Drop back to recursive chunking |
| Agentic chunking blows past its compute budget on a large corpus | Per-document analysis is repeated on every run instead of cached | Cache strategy decisions, or run the analysis at the collection level rather than per document |
These pitfalls can help you identify when a chunking strategy needs to be adjusted for better retrieval and efficiency.
Wrapping Up #
Choosing a chunking strategy starts with understanding the structure of your documents and the retrieval task they need to support. Factors such as document structure, context dependencies, information density, retrieval requirements, and cost all shape that choice.
Use the decision tree to identify a suitable starting point, then validate it with your own queries and retrieval results. Start with the simplest strategy that fits your needs, and introduce more advanced approaches only when they address a specific retrieval challenge. The best results come from matching chunk boundaries to the information you need to retrieve, while keeping the added complexity proportional to the value it provides.