How to Decide Chunk Size in Any Project: Complete Interview Guide

This guide covers the decision-making framework that interviewers expect you to explain, with real-world trade-offs and problem-solving approach.

What Interviewer Wants to Hear:

"Chunk size is NOT a fixed number - it's a design decision based on multiple factors: document type, retrieval precision, token budget, LLM context window, and latency requirements. I use a systematic approach: analyze constraints → model requirements → test with metrics → iterate."

Interview Question: “How would chunk size differ for legal documents vs news articles?”

Answer Template:

Legal documents (1000-2000 words):- Why: A clause or section is an indivisible unit of meaning- Risk: Breaking mid-clause creates ambiguity in retrieval- Example: Contract terms must be complete to be accurate
News articles (400-800 words):- Why: Readers expect paragraph-level information- Benefit: Smaller chunks allow precise topic retrieval- Trade-off: May need overlap to connect ideas

Ask yourself these questions:

1. PRECISION REQUIREMENT   □ High precision needed? → SMALLER chunks (400-600 words)     Example: Legal discovery, medical diagnosis      □ Moderate precision? → MEDIUM chunks (700-1200 words)     Example: Technical support, FAQ systems      □ Broad context OK? → LARGER chunks (1500-2500 words)     Example: General knowledge Q&A
2. RETRIEVAL TYPE   □ Exact match needed? → SMALLER chunks     Example: "Find this specific clause"      □ Semantic match? → MEDIUM chunks     Example: "Find information about refunds"      □ Topic match? → LARGER chunks     Example: "Explain machine learning basics"
3. LATENCY REQUIREMENTS   □ Real-time (< 500ms)? → SMALLER chunks     (Faster embedding, faster search)      □ Sub-second (< 2s)? → MEDIUM chunks      □ Can tolerate delay? → LARGER chunks     (More context means better answers)
4. COST SENSITIVITY   □ Budget tight? → SMALLER chunks     (Fewer embeddings to compute)      □ Can spend? → LARGER chunks     (More embeddings = better precision)

Interview Answer Example:

"For a chatbot answering customer support tickets, I'd analyze:- Precision: Medium-high (accuracy matters)- Retrieval: Semantic (users phrase questions differently)- Latency: < 2 seconds (acceptable for chat)- Cost: Moderate budget
This suggests 600-1000 word chunks with 15% overlap."
CONTEXT_BUDGET = LLM_CONTEXT_WINDOW * 0.6  # Reserve 60% for safety

Interview Answer:

"Given Claude's 200K token window, I'd allocate:- 60% for context (120K tokens) - 15% for prompt (30K tokens)- 15% for response (30K tokens)- 10% safety buffer (20K tokens)
If each 2000-char chunk ≈ 500 tokens, I can safely fit 10-15 chunks. Working backward: 15 chunks * 2000 chars = 30,000 chars maximum context.
For better retrieval quality, I'd use parent-child chunking where top-5 most relevant child chunks expand to their parent chunks for full context."
EMBEDDING_LIMITS = {    "OpenAI (text-embedding-3-large)": "8,191 tokens (~6,000 words)",    "Anthropic (own embeddings)": "varies",    "Cohere": "1,024 tokens (safe limit)",    "Local models (ONNX)": "512-2048 tokens typically",}

Interview Question: How does embedding model capacity affect your chunk size decision?

Answer:

"Embedding models have token limits. For example, Cohere's model safely handles 1024 tokens. Since 1 token ≈ 4 characters:- 1024 tokens ≈ 4000 characters max
Even if I want 2000-char chunks for my use case, I must verify the embedding model can handle it. If it can't, I either:1. Use a more capable embedding model (OpenAI's 8K model)2. Reduce chunk size (1500-2000 chars)3. Split chunks: embed smaller, but retrieve with parent for context
The embedding model is often the limiting factor."
class ChunksizeMetrics:    """    What interviewers expect you to measure:    """        def __init__(self):        self.metrics = {            # Retrieval Quality            "Precision@K": "Of top-K results, how many relevant?",            "Recall@K": "Of all relevant docs, how many in top-K?",            "NDCG@K": "Normalized Discounted Cumulative Gain (ranking quality)",            "MRR": "Mean Reciprocal Rank (position of first relevant result)",                        # Cost & Performance            "Embedding_Time": "Seconds to embed all chunks",            "Search_Latency": "Seconds to find top-K results",            "Storage_Size": "GB needed to store all embeddings",            "Cost_Per_Query": "$ cost (API calls for embeddings)",                        # Answer Quality            "Hallucination_Rate": "% of made-up information in answers",            "Citation_Accuracy": "% of citations point to correct source",            "Context_Coverage": "% of relevant information in context",            "Token_Efficiency": "Useful info per token in context"        }

Interviewer Question: How would you evaluate if your chunk size is optimal?

Answer:

"I'd run experiments with 3 chunk sizes: small (400 words), medium (800 words), large (1500 words).
For each, I measure:
1. RETRIEVAL QUALITY (does RAG find relevant info?)   - Precision@5: "Of top 5 results, are they relevant?"   - Recall@10: "Of all relevant documents, how many appear?"   - Target: Precision > 0.8, Recall > 0.7
bash
2. COST & LATENCY (can we afford it?)   - Embedding cost: $ per 1M chunks   - Search latency: milliseconds   - Storage: GB for all embeddings   - Target: <100ms latency, <1KB per chunk metadata
3. ANSWER QUALITY (does LLM generate good responses?)   - Hallucination rate: Manual review of 50 samples   - Citation accuracy: Does answer reference correct chunks?   - Token efficiency: Useful info per token   - Target: <5% hallucinations, >90% citations accurate
Then I pick the size that balances these metrics best."
class ChunkSizeExperiment:    """    Structure your testing like a real data scientist    """        CHUNK_SIZES = [400, 600, 800, 1000, 1500]  # Words    EVALUATION_SET = 100  # Test queries        def run_experiment(self):        results = {}                for size in self.CHUNK_SIZES:            # Step 1: Create chunks of this size            chunks = create_chunks(text, size_words=size)                        # Step 2: Embed all chunks            embeddings = embed_all(chunks)            storage.index(embeddings)                        # Step 3: Run test queries            metrics = {                "precision_5": 0.0,                "recall_10": 0.0,                "latency_ms": 0.0,                "cost_usd": 0.0,                "hallucination_rate": 0.0,            }                        for query in EVALUATION_SET:                results_top5 = retrieve(query, k=5)                # Evaluate relevance, measure latency, etc.                        results[size] = metrics                return self.analyze_results(results)        def analyze_results(self, results):        """        Present findings professionally        """        print("""        CHUNK SIZE ANALYSIS RESULTS        ============================                Size  | Precision | Recall | Latency | Cost  | Halluc.        ------|-----------|--------|---------|-------|--------        400w  | 0.85      | 0.72   | 45ms    | $0.8  | 3.2%        600w  | 0.88      | 0.78   | 50ms    | $1.2  | 2.1%        800w  | 0.86      | 0.82   | 55ms    | $1.6  | 1.8%  ⭐ BEST        1000w | 0.82      | 0.80   | 65ms    | $2.0  | 1.5%        1500w | 0.79      | 0.75   | 80ms    | $3.0  | 2.0%                RECOMMENDATION: 800 words        - Highest Recall (0.82)        - Good Precision (0.86)        - Reasonable latency (55ms)        - Manageable cost        - Lowest hallucination rate (1.8%)        """)

Interview Presentation:

"I'd create a simple experiment with 5 chunk sizes and run 100 test queries. For each size, I measure precision, recall, latency, and cost. Based on the results, 800-word chunks provide the best balance: high recall (fewer missed docs), good precision (fewer irrelevant results), and reasonable cost.
The key insight: larger chunks give better recall (more context), but smaller chunks give better precision (less noise). 800 words is the sweet spot."
class DomainSpecificChunkSizes:    """    Interview tip: Show you understand domain context    """        DOMAINS = {        "LEGAL": {            "size": "1000-2000 words",            "reason": "Sections/clauses are legal units",            "example": "Contract clause must be complete",            "key_metric": "Precision > Recall (accuracy critical)",            "overlap": "20% (preserve clause boundaries)",        },                "MEDICAL": {            "size": "500-1000 words",            "reason": "Patient outcomes depend on complete context",            "example": "Symptoms + test results + diagnosis",            "key_metric": "Recall > Precision (miss nothing)",            "overlap": "20% (connect symptoms to outcomes)",        },                "E-COMMERCE": {            "size": "400-800 words",            "reason": "Product info is naturally separated",            "example": "Product specs, reviews, shipping info",            "key_metric": "Speed (real-time product search)",            "overlap": "10% (less critical)",        },                "CUSTOMER_SUPPORT": {            "size": "600-1000 words",            "reason": "Q&A pairs with explanation",            "example": "Question + answer + examples",            "key_metric": "User satisfaction (answers must be complete)",            "overlap": "15%",        },                "TECHNICAL_DOCS": {            "size": "500-1000 words",            "reason": "API docs, parameters need to stay together",            "example": "Function signature + params + examples",            "key_metric": "Accuracy (wrong example breaks code)",            "overlap": "15%",        },                "NEWS/MEDIA": {            "size": "400-600 words",            "reason": "Articles are already well-written units",            "example": "One news story = one natural unit",            "key_metric": "Latency (real-time relevance)",            "overlap": "10%",        },    }

Interview Answer Example:

"For a medical chatbot vs an e-commerce bot, chunk sizing would be very different:
MEDICAL (diagnosing symptoms):- Chunk size: 700-1000 words- Why: Symptoms, tests, diagnosis, treatment must be together- Metric: Optimize for RECALL (don't miss anything)- Overlap: 20% (very important - connect related symptoms)
E-COMMERCE (product recommendation):- Chunk size: 500-800 words  - Why: Products are naturally separate; specs are self-contained- Metric: Optimize for SPEED (<100ms) and cost- Overlap: 10% (less critical)
The key difference: Medical prioritizes completeness; e-commerce prioritizes speed."
❌ "We just use 512-token chunks like everyone else"   → Shows no independent thinking
❌ "Bigger chunks are always better"   → Ignores retrieval precision vs recall trade-off
❌ "We never tested different chunk sizes"   → Suggests no systematic approach
❌ "Chunk size doesn't matter much"   → Shows ignorance of its impact
✅ Instead say:   "We systematically tested 5 different chunk sizes on our    evaluation set, measuring precision, recall, latency, and cost.    Based on the results, 800-word chunks provided the best balance    for our use case."

Answer:

"Good question. This is where recursive chunking + parent-child hierarchy becomes critical.
Approach:1. DETECT boundaries: Identify natural sections, paragraphs, sentences2. RECURSIVE splitting: Try to split at paragraph level first3. FALL BACK gracefully: If paragraph > chunk_size, split by sentences4. PRESERVE context: Use parent chunks to keep full section
Example: If I want 800-word chunks but a legal clause is 1200 words:- Don't: Break the clause (loses meaning)- Do: Keep clause as one parent chunk      Split into 2-3 child chunks (for embedding)      Retrieve child chunks + expand to full parent (for LLM)
This preserves semantic integrity while optimizing for retrieval."

Answer:

"Variable-length documents need adaptive chunking:
1. ANALYZE document length   - If < 2000 words: Use as single chunk   - If 2000-10000 words: Split into 3-5 chunks   - If > 10000 words: Use hierarchical chunking
2. USE overlap intelligently   - Short docs: 10% overlap (minimal redundancy)   - Long docs: 20% overlap (ensure continuity)   - Large docs: 25% overlap (bridge multiple chunks)
3. IMPLEMENT dynamic sizing   chunks = []   if doc_length < 2000:       chunks = [entire_doc]   else:       base_size = doc_length / (num_sections * 0.8)       chunks = recursive_split(doc, size=base_size)
This ensures consistency while respecting document structure."

Answer:

"Mixed content requires special handling:
STRATEGY:1. DETECT content type   if is_table:       chunk_size = 1000_chars  (tables are dense)   elif is_code:       chunk_size = 500_chars   (code needs precision)   else:       chunk_size = 2000_chars  (normal text)
2. KEEP units intact   - Table: Never split a row across chunks   - Code: Never split a function/class   - Text: Split at paragraph boundary
3. ADD metadata   Each chunk stores:   - content_type: 'text', 'table', 'code'   - importance: 'high', 'normal', 'low'   - structure: original section/subsection
4. USE in retrieval   Query for code → prioritize code chunks   Query for stats → prioritize table chunks
This ensures retrieval quality across mixed content."

Answer:

"Production data often differs from dev data. I'd implement:
1. MONITORING in production   - Track precision/recall on real queries   - Monitor hallucination rate   - Watch for queries that fail   - Collect user feedback
2. A/B TESTING for changes   Before changing chunk size, run experiment:   - Control: Current chunk size (95% traffic)   - Test: New chunk size (5% traffic)   - Measure: Impact on user satisfaction, metrics
3. GRADUAL ROLLOUT   - Week 1: Test new size with 5% queries   - Week 2: Expand to 20% if metrics good   - Week 3: Full rollout or rollback
4. FALLBACK PLAN   If new size performs worse:   - Immediately revert to previous size   - Investigate issue   - Re-test with adjusted size
This ensures production stability while allowing optimization."

Print this before your interview! Interviewers love when you reference a framework.

CHUNK SIZE DECISION MATRIX==========================
SMALL (400-600w) | MEDIUM (800-1200w) | LARGE (1500-2500w)Precision/Recall Trade  | High Precision   | Balanced           | High RecallLatency                 | Fast (<50ms)     | Moderate (50-80ms) | Slow (>80ms)Cost                    | Low              | Medium             | HighSemantic Preservation   | At Risk          | Good               | ExcellentEmbedding Model Load    | Low              | Medium             | HighLLM Context Budget      | Large margin     | Comfortable        | TightBest For                | E-commerce       | Support/FAQ        | Legal/Medical                        | News             | Tech docs          | Research
YOUR USE CASE:- Domain: _______- Priority: Speed? Cost? Accuracy?- Document length: Short / Medium / Long- Precision vs Recall: Which matters more?
RECOMMENDED SIZE: _______ words

When asked “How do you decide chunk size?”, end with this:

"In summary, it's a three-step process:
1. ANALYZE constraints   - Document type (legal vs news vs code)   - Precision vs recall needs   - Latency and cost budgets   - Model capabilities
2. EMPIRICAL testing   - Test 3-5 different sizes   - Measure precision, recall, latency, cost   - Evaluate hallucination rate   - Select size that balances trade-offs
3. PRODUCTION monitoring   - Track real-world metrics   - Run A/B tests before changes   - Implement fallback plan   - Iterate based on user feedback
The key insight: There's no universal answer. It's a design decision based on YOUR specific constraints and priorities.
For this project, I would recommend [size] words because [specific reasons related to their domain]."

“I’ve researched this topic through:”

How to Decide Chunk Size in Any Project: Complete Interview Guide was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.