Imagine I give you a whole pizza and say:
“Eat it.”
You look at it and think, Sure, I can eat it.
But now imagine I give you the same pizza without cutting it.
Can you eat it comfortably?
Not really.
You would probably cut it into smaller pieces first.
And that is exactly what we do with large documents before giving them to an AI system.
A 200-page PDF, a 50,000-line documentation file, or a huge company knowledge base is useful to us as a complete document.
But for a Retrieval-Augmented Generation (RAG) system, giving the entire document to the retrieval system is usually not the best idea.
So we break it down.
Document
↓
Chunks
↓
Embeddings
↓
Vector Database
↓
Retrieval
↓
LLM
↓
Answer
That process of breaking a large document into smaller, meaningful pieces is called Chunking.
And here is the interesting part:
Chunking is not simply “splitting text into 500 characters.”
It is a retrieval design decision.
The wrong chunk can make the right information difficult to retrieve.
Let's say we have this document:
Company Employee Handbook
Chapter 1: Leave Policy
Employees receive 20 paid leaves every year...
Chapter 2: Work From Home
Employees can work remotely two days per week...
Chapter 3: Insurance
Employees are eligible for health insurance...
Now a user asks:
“How many days can I work from home?”
We don't need the entire employee handbook.
We need the small section containing the Work From Home Policy.
So ideally:
DOCUMENT
│
┌────────────┼────────────┐
↓ ↓ ↓
Chunk 1 Chunk 2 Chunk 3
│ │ │
Leave Policy WFH Policy Insurance
│
↓
Relevant Chunk
The goal is simple:
Retrieve the smallest useful piece of information without destroying its meaning.
There are two opposite mistakes.
Imagine:
┌──────────────────────────────┐
│ Leave Policy │
│ Work From Home │
│ Insurance │
│ Salary │
│ Performance Review │
│ Travel Policy │
│ ... │
└──────────────────────────────┘
You ask:
“How many leaves do I get?”
The answer exists somewhere inside the chunk.
But so does a lot of unrelated information.
The retriever has brought the answer plus noise.
Now imagine:
Chunk 1:
Employees are eligible for
Chunk 2:
20 days of paid leave
Chunk 3:
every financial year.
The original document contained a complete thought.
Our chunking destroyed it.
The information exists.
But the meaning is fragmented.
This gives us the first important principle:
Chunking is a balance between context and precision.
Too large:
More context
↓
More noise
Too small:
More precision
↓
Less context
The goal is somewhere in between.
A typical RAG pipeline looks like this:
ORIGINAL DOCUMENT
│
▼
┌─────────────┐
│ Chunking │
└──────┬──────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
Chunk 1 Chunk 2 Chunk 3
│ │ │
▼ ▼ ▼
Embedding Embedding Embedding
│ │ │
└────────────┼────────────┘
▼
Vector Database
│
│ User Query
▼
Query Embedding
│
▼
Retrieval
│
▼
Relevant Chunks
│
▼
LLM
│
▼
Answer
There is one important thing to notice:
Chunking decides what the embedding model actually gets to understand.
If you change the chunks, you change the embeddings.
If you change the embeddings, you can change retrieval.
And if retrieval changes, the context given to the LLM changes.
So chunking is not just preprocessing.
It is part of your retrieval architecture.
This is probably one of the most misunderstood parts of chunking.
Someone might ask:
“My LLM supports 128K tokens. Should I create 8K-token chunks?”
Not necessarily.
Your LLM's context window is only one part of the equation.
Your chunk size depends on several things:
CHUNK SIZE
│
┌────────────────┼────────────────┐
↓ ↓ ↓
Embedding Model Retrieval LLM
Input Limit Requirements Context Budget
│ │ │
└────────────────┼────────────────┘
│
▼
Practical Chunk Size
And there are two more factors that matter:
So the decision becomes:
Document Structure
+
Embedding Model Limit
+
Retrieval Requirements
+
LLM Context Budget
+
Question Type
↓
Candidate Chunk Sizes
↓
Evaluation
↓
Final Chunk Size
This is why there is no universal:
chunk_size = 500
number.
Don't start with “What chunk size does everyone use?” Start with “What information am I trying to retrieve?”
There is no universally best chunking strategy.
A legal contract, a Markdown documentation page, and a Python codebase don't contain information in the same structure.
So why would we split them in exactly the same way?
LangChain provides different text splitters, and its documentation recommends RecursiveCharacterTextSplitter
as a strong starting point for generic text.
Let's group the important approaches.
The simplest approach:
Every 1000 characters → new chunk
Example:
from langchain_text_splitters import CharacterTextSplitter
splitter = CharacterTextSplitter(
chunk_size=1000,
chunk_overlap=100
)
chunks = splitter.split_text(text)
The advantage?
Simple and predictable.
The problem?
It doesn't necessarily care about meaning.
It can split:
at the end of a paragraph
or:
in the middle of a sentence.
So fixed-size splitting is useful for simple, predictable data, but it is rarely the only strategy worth considering.
This is usually the best starting point for general text.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
chunks = splitter.split_text(text)
Conceptually, the splitter tries to preserve larger natural boundaries before breaking the text further:
Paragraph
↓
Newline
↓
Space
↓
Character
So instead of immediately cutting through a sentence, it tries to preserve meaningful blocks first.
Document
│
├── Paragraph
│ │
│ └── fits? → keep together
│
├── Paragraph
│ │
│ └── too large?
│ ↓
│ split further
│
└── ...
That's why it is such a useful generic starting point.
When you don't know where to start, start simple. Then measure.
Characters and tokens are not the same thing.
For example:
1000 characters ≠ 1000 tokens
The exact relationship depends on the tokenizer, language, and text.
If your downstream system has strict token constraints, token-aware splitting becomes useful.
from langchain_text_splitters import TokenTextSplitter
splitter = TokenTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = splitter.split_text(text)
Token-based splitting becomes particularly useful when you need precise control over how much tokenized content enters the embedding or downstream pipeline.
Sometimes the document already tells us how it should be chunked.
Consider Markdown:
## Login
Information about login...
## Password Reset
Information about resetting passwords...
## Refunds
Information about refunds...
Why destroy that structure?
Instead, preserve it.
The same idea applies to:
For example, a chunk can retain:
Document: API Documentation
Section: Authentication
Subsection: Password Reset
Content:
To reset your password...
Now the chunk contains both:
information + context
That's much more useful for retrieval.
Code is not prose.
Consider:
class PaymentService:
def process_payment(self):
...
Randomly splitting every 500 characters can easily separate:
class
↓
method
↓
implementation
which destroys useful relationships.
For code, you want to preserve programming structure where possible:
Repository
↓
Class
↓
Method
↓
Logical Block
LangChain provides language-aware splitting for multiple programming languages.
So for code:
Respect the syntax before respecting the character count.
Now we move from:
“Where should I split the characters?”
to:
“Where does the meaning change?”
Consider:
The company was founded in 1998.
It started with five employees.
The company launched its first product in 2001.
Revenue crossed $10M in 2005.
There is a natural semantic transition between these ideas.
Semantic chunking attempts to identify those changes rather than blindly following fixed character boundaries.
This can be useful for highly topic-driven documents.
But there is a trade-off.
It can introduce:
So don't automatically assume:
Semantic = Better
Sometimes:
Simple + predictable + evaluated > sophisticated + unevaluated.
Now we reach the famous:
chunk_size=1000
chunk_overlap=200
But what do these numbers actually mean?
Chunk size defines how much content goes into one chunk.
For example:
chunk_size = 1000
means the splitter attempts to create chunks around that size according to the unit it uses.
That unit could be:
For example, RecursiveCharacterTextSplitter
measures characters by default.
Suppose the document says:
Machine learning models require training data.
The quality of this data directly affects model performance.
Good data leads to better generalization.
If we split aggressively:
Chunk 1:
Machine learning models require training data.
Chunk 2:
The quality of this data directly affects model performance.
The relationship between the two chunks can become weaker.
Overlap creates a bridge.
Chunk 1
┌───────────────────────────────┐
│ Machine learning models... │
│ Training data affects... │
└───────────────┬───────────────┘
│
overlap
│
▼
┌───────────────────────────────┐
Chunk 2 │ Training data affects... │
│ Good data leads to... │
└───────────────────────────────┘
So:
Overlap protects context at the boundary.
But more overlap is not automatically better.
Too much overlap means:
More overlap
↓
More chunks
↓
More embeddings
↓
More storage
↓
More duplicate retrieval
↓
More context repetition
↓
Higher cost
Use overlap when it helps preserve meaning across boundaries.
Let's build a simple example.
Install the splitter package:
pip install -U langchain-text-splitters
Then:
from langchain_text_splitters import RecursiveCharacterTextSplitter
text = """
Chunking is important for Retrieval Augmented Generation.
Large documents are difficult to retrieve efficiently.
By breaking documents into smaller meaningful pieces,
we can retrieve only the information required to answer
a user's question.
"""
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = splitter.split_text(text)
for i, chunk in enumerate(chunks):
print(f"Chunk {i}")
print(chunk)
print("-" * 50)
The flow is:
Original Text
│
▼
RecursiveCharacterTextSplitter
│
┌────┼────┐
↓ ↓ ↓
C1 C2 C3
│ │ │
└────┼────┘
↓
Embeddings
For token-controlled pipelines:
from langchain_text_splitters import TokenTextSplitter
splitter = TokenTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = splitter.split_text(text)
And for structured documents, choose the splitter that understands that structure rather than flattening everything into plain text.
Here's the question that actually matters:
Which chunking method should I use?
Don't choose based on popularity.
Choose based on your data.
| Document Type | Good Starting Strategy |
|---|---|
| General text | Recursive |
| Markdown | Header / structure-aware |
| HTML | HTML-aware |
| Source code | Language-aware |
| JSON | Structure-aware |
| Highly topic-driven text | Semantic |
| Strict token constraints | Token-based |
| Legal / structured documents | Structure-aware |
| Tables | Preserve table structure |
Think about it like this:
What is my data?
│
┌────────────────┼────────────────┐
↓ ↓ ↓
Plain Text Structured Code
│ │ │
↓ ↓ ↓
Recursive Structure- Language-
aware aware
│ │ │
└────────────────┼────────────────┘
↓
Check Embedding Model
↓
Choose Candidates
↓
Evaluate
And that last step is important.
Choosing a chunking strategy is a hypothesis.
Evaluation tells you whether the hypothesis was correct.
This is where many chunking tutorials stop.
They shouldn't.
Because the real question isn't:
“Did my document split successfully?”
It is:
“Did splitting improve retrieval?”
A chunk can look perfectly reasonable to a human and still perform badly in a retrieval system.
So we need an evaluation dataset.
Suppose we're building an employee-policy chatbot.
Create questions such as:
Q1: How many annual leaves does an employee get?
Q2: How many days can employees work remotely?
Q3: What is the maternity leave duration?
Q4: What happens to unused leaves?
Q5: What is the resignation notice period?
For each question, know where the answer actually exists.
For example:
Question:
How many annual leaves does an employee get?
Expected Source:
Employee Handbook
→ Leave Policy
→ Section 2.1
Now we have something measurable.
Suppose the correct chunk is:
Chunk 17
Our retriever returns:
Top 5:
Chunk 91
Chunk 43
Chunk 17 ← Correct
Chunk 52
Chunk 8
The correct chunk appeared in the top five.
That's useful.
Now we can measure different aspects of retrieval quality.
Recall@K asks:
Did the relevant chunk appear anywhere in the top K results?
For example:
Top 5:
Chunk 91
Chunk 43
Chunk 17 ← Relevant
Chunk 52
Chunk 8
The relevant chunk is present.
Therefore:
Recall@5 = 1
or 100% for this query.
Across many queries:
Recall@5 =
Queries where relevant information
appeared in top 5
───────────────────────────────────
Total queries
So if:
90 / 100
queries retrieved the correct information within the top 5:
Recall@5 = 90%
In simple terms:
Recall asks: “Did I find it?”
Precision@K asks a different question:
“How many of the retrieved results were actually relevant?”
Suppose:
Top 5:
Chunk 17 → Relevant
Chunk 21 → Relevant
Chunk 42 → Irrelevant
Chunk 63 → Irrelevant
Chunk 91 → Irrelevant
Then:
Precision@5 = 2 / 5
= 40%
So:
Recall → Did I find the answer?
Precision → How much irrelevant information did I retrieve?
This distinction becomes especially important when large chunks contain multiple unrelated topics.
Now imagine two systems.
Top 5:
1. Correct ✓
2. Wrong
3. Wrong
4. Wrong
5. Wrong
Top 5:
1. Wrong
2. Wrong
3. Correct ✓
4. Wrong
5. Wrong
Both systems found the correct answer within the top 5.
So their Recall@5 is the same.
But are they equally good?
Not really.
System A put the correct result first.
That's where MRR — Mean Reciprocal Rank becomes useful.
For one query:
MRR = 1 / rank of first relevant result
So:
Correct at rank 1
→ 1 / 1
→ 1.0
Correct at rank 2
→ 1 / 2
→ 0.5
Correct at rank 3
→ 1 / 3
→ 0.33
Correct at rank 5
→ 1 / 5
→ 0.20
For multiple queries, we take the average of these reciprocal ranks.
For example:
Query 1 → Correct at #1 → 1.00
Query 2 → Correct at #2 → 0.50
Query 3 → Correct at #3 → 0.33
MRR = (1.00 + 0.50 + 0.33) / 3
≈ 0.61
So:
Recall tells you whether you found the answer. MRR tells you how high you ranked the first correct answer.
This is particularly useful when the order of retrieved results matters.
A RAG system has multiple layers:
User Question
│
▼
Retrieval
│
▼
Retrieved Chunks
│
▼
LLM
│
▼
Answer
Therefore, evaluate both retrieval and final answers.
Did we retrieve the right information?
Useful metrics:
Did the model actually use that information correctly?
Useful dimensions:
Answer Correctness
Did the model produce the correct answer?
Answer Relevance
Did the answer actually address the user's question?
Faithfulness / Groundedness
Is the answer supported by the retrieved context?
This distinction matters.
Good retrieval does not automatically mean a good answer.
And:
A good-looking answer does not automatically mean good retrieval.
Let's say we have:
1,000 documents
200 evaluation questions
We test three configurations.
Recursive
chunk_size = 256
overlap = 32
Recursive
chunk_size = 512
overlap = 64
Recursive
chunk_size = 1024
overlap = 128
Now measure:
| Configuration | Recall@5 | Precision@5 | MRR | Answer Correctness |
|---|---|---|---|---|
| 256 / 32 | 82% | 76% | 0.71 | 84% |
| 512 / 64 | 91% | 83% | 0.86 | 92% |
| 1024 / 128 | 93% | 61% | 0.78 | 87% |
Which one wins?
Probably:
512 / 64
Why?
The 1024 configuration has slightly higher Recall@5.
But it retrieves significantly more irrelevant information and ranks the relevant information less effectively.
So the final answer quality also drops.
This gives us another important principle:
The largest chunk is not necessarily the best chunk.
And:
The goal isn't to maximize one metric. It's to find the best trade-off for your application.
Let's say our chunk is:
Employees can work remotely two days per week.
That's useful.
But this is better:
Document: Employee Handbook
Section: Work From Home
Department: HR
Year: 2026
Content:
Employees can work remotely two days per week.
Why?
Because the content tells us what the information says.
Metadata tells us where it came from.
Metadata can also enable filtering:
department = HR
document = Employee Handbook
year = 2026
Then semantic retrieval can happen over a much more relevant subset.
So:
Good chunking tells you what the text says. Good metadata tells you where it belongs.
Tutorial says 500
↓
I use 500
That's not a strategy.
Test multiple values against your actual data.
Your LLM may support 128K tokens.
Your embedding model has its own input constraints.
Always check:
Embedding model
↓
Input/token limit
↓
Tokenizer
↓
Actual chunk size
A legal contract and a Python codebase are not the same thing.
Use the structure already present in the data.
More overlap doesn't automatically mean more context.
It can mean:
More duplication
↓
More embeddings
↓
More storage
↓
More retrieval noise
↓
Higher cost
Imagine:
| Product | Price | Discount |
|---------|-------|----------|
| A | 100 | 10% |
| B | 200 | 15% |
If you blindly split this structure, you can lose relationships between columns and values.
For structured data:
Preserve structure before splitting aggressively.
If I were starting a new RAG system tomorrow, I wouldn't immediately build a complicated semantic chunking pipeline.
I'd start simple.
PDF?
Markdown?
HTML?
Code?
Legal documents?
Tables?
Document
↓
Headers
↓
Sections
↓
Paragraphs
↓
Sentences
Don't destroy structure that already exists.
Ask:
What is the model's input limit?
What tokenizer does it use?
How does it behave with longer inputs?
Remember:
The embedding model is part of the chunk-size decision.
For example:
Plain text
↓
Recursive
Markdown
↓
Header-aware
Code
↓
Language-aware
Highly topic-driven
↓
Semantic
For example:
256
512
768
1024
with sensible overlap values.
Don't treat these as universal recommendations.
They are simply candidate configurations to test.
Create questions based on what your users actually ask.
Question
↓
Expected Source
↓
Expected Information
Track:
Recall@K
Precision@K
MRR
Track:
Answer Correctness
Answer Relevance
Faithfulness / Groundedness
And where useful, also track:
Latency
Token Usage
Embedding Cost
Storage Cost
Suppose:
Recursive Chunking
performs almost as well as:
Semantic Chunking
but is cheaper, faster, and easier to maintain.
Then:
Take the simpler solution.
The best chunking strategy isn't the most complicated one. It's the one that performs well on your data.
Put everything together:
START
│
▼
Understand Data
│
▼
Preserve Data Structure
│
▼
Check Embedding Model
│
▼
Understand User Questions
│
▼
Choose Initial Strategy
│
▼
Choose Chunk Size Range
│
▼
Choose Overlap
│
▼
Build Evaluation Dataset
│
▼
Run Retrieval Tests
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Recall@K Precision@K MRR
│ │ │
└─────────────┼─────────────┘
↓
Evaluate RAG Answers
│
▼
Correctness / Relevance /
Groundedness
│
▼
Compare Results
│
▼
Tune and Repeat
│
▼
Production
The important thing is that chunking doesn't end when the document is split.
It ends when you know those chunks are helping your retrieval system.
Let's go back to our pizza.
You don't eat a whole pizza in one bite.
You cut it.
But you also don't cut it into 1000 tiny pieces.
Because then you have created another problem.
Chunking works the same way.
Too Large
↓
Too Much Noise
Too Small
↓
Lost Context
Just Right
↓
Better Retrieval
↓
Better Context
↓
Better Answers
And the "just right" size is not a universal number.
It depends on:
your document structure + your questions + your embedding model + your retrieval strategy + your LLM context budget.
So don't ask:
“What is the best chunk size?”
Ask:
“What is the best chunk size for my data, my embedding model, and my retrieval problem?”
Because ultimately:
Chunking isn't about making documents smaller.
It's about making knowledge retrievable.
And that's the real job of chunking.
Cut the document enough to retrieve what matters — but not so much that you lose why it matters.