{"slug": "how-to-decide-chunk-size-in-any-project-complete-interview-guide", "title": "How to Decide Chunk Size in Any Project: Complete Interview Guide", "summary": "A new interview guide outlines a systematic framework for determining chunk size in retrieval-augmented generation projects, emphasizing that chunk size is a design decision based on document type, retrieval precision, token budget, LLM context window, and latency. The guide provides specific ranges—legal documents 1000-2000 words, news articles 400-800 words—and includes a calculation example using Claude 3 Sonnet's 200K token window, allocating 60% for context, 15% for prompt, 15% for response, and 10% safety buffer, resulting in 10-15 chunks of 2000 characters each. It also lists embedding model limits, such as OpenAI's text-embedding-3-large at 8,191 tokens and Cohere at 1,024 tokens, which constrain chunk size before embedding.", "body_md": "This guide covers the decision-making framework that interviewers expect you to explain, with real-world trade-offs and problem-solving approach.\n\n**What Interviewer Wants to Hear:**\n\n```\n\"Chunk size is NOT a fixed number - it's a design decision based on multiple factors: document type, retrieval precision, token budget, LLM context window, and latency requirements. I use a systematic approach: analyze constraints → model requirements → test with metrics → iterate.\"\n```\n\n**Interview Question:** *“How would chunk size differ for legal documents vs news articles?”*\n\n**Answer Template:**\n\n```\nLegal documents (1000-2000 words):- Why: A clause or section is an indivisible unit of meaning- Risk: Breaking mid-clause creates ambiguity in retrieval- Example: Contract terms must be complete to be accurate\nNews articles (400-800 words):- Why: Readers expect paragraph-level information- Benefit: Smaller chunks allow precise topic retrieval- Trade-off: May need overlap to connect ideas\n```\n\nAsk yourself these questions:\n\n```\n1. PRECISION REQUIREMENT   □ High precision needed? → SMALLER chunks (400-600 words)     Example: Legal discovery, medical diagnosis      □ Moderate precision? → MEDIUM chunks (700-1200 words)     Example: Technical support, FAQ systems      □ Broad context OK? → LARGER chunks (1500-2500 words)     Example: General knowledge Q&A\n2. RETRIEVAL TYPE   □ Exact match needed? → SMALLER chunks     Example: \"Find this specific clause\"      □ Semantic match? → MEDIUM chunks     Example: \"Find information about refunds\"      □ Topic match? → LARGER chunks     Example: \"Explain machine learning basics\"\n3. LATENCY REQUIREMENTS   □ Real-time (< 500ms)? → SMALLER chunks     (Faster embedding, faster search)      □ Sub-second (< 2s)? → MEDIUM chunks      □ Can tolerate delay? → LARGER chunks     (More context means better answers)\n4. COST SENSITIVITY   □ Budget tight? → SMALLER chunks     (Fewer embeddings to compute)      □ Can spend? → LARGER chunks     (More embeddings = better precision)\n```\n\n**Interview Answer Example:**\n\n```\n\"For a chatbot answering customer support tickets, I'd analyze:- Precision: Medium-high (accuracy matters)- Retrieval: Semantic (users phrase questions differently)- Latency: < 2 seconds (acceptable for chat)- Cost: Moderate budget\nThis suggests 600-1000 word chunks with 15% overlap.\"\n# Key calculation interviewers expect:\nCONTEXT_BUDGET = LLM_CONTEXT_WINDOW * 0.6  # Reserve 60% for safety\n# Example with Claude 3 Sonnet (200K tokens)total_tokens = 200_000safe_budget = total_tokens * 0.6  # 120,000 tokensreserved_for_prompt = 20_000 tokensreserved_for_response = 10_000 tokensavailable_for_context = 90_000 tokens\n# Now work backwards from chunks:tokens_per_1000_chars = 250  # Rough estimateavailable_chars = 90_000 / 250 * 1000  # ~360,000 characters\n# If chunk_size = 2000 chars, how many chunks can we fit?max_chunks = 360_000 / 2000  # ~180 chunks# But typically use only top-5 to top-10 chunksreasonable_budget = 10 * 2000 # 20,000 chars = 5,000 tokens\n```\n\n**Interview Answer:**\n\n```\n\"Given Claude's 200K token window, I'd allocate:- 60% for context (120K tokens) - 15% for prompt (30K tokens)- 15% for response (30K tokens)- 10% safety buffer (20K tokens)\nIf each 2000-char chunk ≈ 500 tokens, I can safely fit 10-15 chunks. Working backward: 15 chunks * 2000 chars = 30,000 chars maximum context.\nFor better retrieval quality, I'd use parent-child chunking where top-5 most relevant child chunks expand to their parent chunks for full context.\"\n# Most embedding models have input limitations:\nEMBEDDING_LIMITS = {    \"OpenAI (text-embedding-3-large)\": \"8,191 tokens (~6,000 words)\",    \"Anthropic (own embeddings)\": \"varies\",    \"Cohere\": \"1,024 tokens (safe limit)\",    \"Local models (ONNX)\": \"512-2048 tokens typically\",}\n# This limits chunk size BEFORE embedding:# If embedding model accepts 1024 tokens max:# That's roughly 750-1000 words or 3000-4000 characters\n# Safe chunk size < embedding_limitCHUNK_SIZE = min(2000_chars, embedding_model_limit)\n```\n\n**Interview Question:** *How does embedding model capacity affect your chunk size decision?*\n\n**Answer:**\n\n```\n\"Embedding models have token limits. For example, Cohere's model safely handles 1024 tokens. Since 1 token ≈ 4 characters:- 1024 tokens ≈ 4000 characters max\nEven if I want 2000-char chunks for my use case, I must verify the embedding model can handle it. If it can't, I either:1. Use a more capable embedding model (OpenAI's 8K model)2. Reduce chunk size (1500-2000 chars)3. Split chunks: embed smaller, but retrieve with parent for context\nThe embedding model is often the limiting factor.\"\nclass ChunksizeMetrics:    \"\"\"    What interviewers expect you to measure:    \"\"\"        def __init__(self):        self.metrics = {            # Retrieval Quality            \"Precision@K\": \"Of top-K results, how many relevant?\",            \"Recall@K\": \"Of all relevant docs, how many in top-K?\",            \"NDCG@K\": \"Normalized Discounted Cumulative Gain (ranking quality)\",            \"MRR\": \"Mean Reciprocal Rank (position of first relevant result)\",                        # Cost & Performance            \"Embedding_Time\": \"Seconds to embed all chunks\",            \"Search_Latency\": \"Seconds to find top-K results\",            \"Storage_Size\": \"GB needed to store all embeddings\",            \"Cost_Per_Query\": \"$ cost (API calls for embeddings)\",                        # Answer Quality            \"Hallucination_Rate\": \"% of made-up information in answers\",            \"Citation_Accuracy\": \"% of citations point to correct source\",            \"Context_Coverage\": \"% of relevant information in context\",            \"Token_Efficiency\": \"Useful info per token in context\"        }\n```\n\n**Interviewer Question:** *How would you evaluate if your chunk size is optimal?*\n\n**Answer:**\n\n```\n\"I'd run experiments with 3 chunk sizes: small (400 words), medium (800 words), large (1500 words).\nFor each, I measure:\n1. RETRIEVAL QUALITY (does RAG find relevant info?)   - Precision@5: \"Of top 5 results, are they relevant?\"   - Recall@10: \"Of all relevant documents, how many appear?\"   - Target: Precision > 0.8, Recall > 0.7\nbash\n2. COST & LATENCY (can we afford it?)   - Embedding cost: $ per 1M chunks   - Search latency: milliseconds   - Storage: GB for all embeddings   - Target: <100ms latency, <1KB per chunk metadata\n3. ANSWER QUALITY (does LLM generate good responses?)   - Hallucination rate: Manual review of 50 samples   - Citation accuracy: Does answer reference correct chunks?   - Token efficiency: Useful info per token   - Target: <5% hallucinations, >90% citations accurate\nThen I pick the size that balances these metrics best.\"\nclass ChunkSizeExperiment:    \"\"\"    Structure your testing like a real data scientist    \"\"\"        CHUNK_SIZES = [400, 600, 800, 1000, 1500]  # Words    EVALUATION_SET = 100  # Test queries        def run_experiment(self):        results = {}                for size in self.CHUNK_SIZES:            # Step 1: Create chunks of this size            chunks = create_chunks(text, size_words=size)                        # Step 2: Embed all chunks            embeddings = embed_all(chunks)            storage.index(embeddings)                        # Step 3: Run test queries            metrics = {                \"precision_5\": 0.0,                \"recall_10\": 0.0,                \"latency_ms\": 0.0,                \"cost_usd\": 0.0,                \"hallucination_rate\": 0.0,            }                        for query in EVALUATION_SET:                results_top5 = retrieve(query, k=5)                # Evaluate relevance, measure latency, etc.                        results[size] = metrics                return self.analyze_results(results)        def analyze_results(self, results):        \"\"\"        Present findings professionally        \"\"\"        print(\"\"\"        CHUNK SIZE ANALYSIS RESULTS        ============================                Size  | Precision | Recall | Latency | Cost  | Halluc.        ------|-----------|--------|---------|-------|--------        400w  | 0.85      | 0.72   | 45ms    | $0.8  | 3.2%        600w  | 0.88      | 0.78   | 50ms    | $1.2  | 2.1%        800w  | 0.86      | 0.82   | 55ms    | $1.6  | 1.8%  ⭐ BEST        1000w | 0.82      | 0.80   | 65ms    | $2.0  | 1.5%        1500w | 0.79      | 0.75   | 80ms    | $3.0  | 2.0%                RECOMMENDATION: 800 words        - Highest Recall (0.82)        - Good Precision (0.86)        - Reasonable latency (55ms)        - Manageable cost        - Lowest hallucination rate (1.8%)        \"\"\")\n```\n\n**Interview Presentation:**\n\n```\n\"I'd create a simple experiment with 5 chunk sizes and run 100 test queries. For each size, I measure precision, recall, latency, and cost. Based on the results, 800-word chunks provide the best balance: high recall (fewer missed docs), good precision (fewer irrelevant results), and reasonable cost.\nThe key insight: larger chunks give better recall (more context), but smaller chunks give better precision (less noise). 800 words is the sweet spot.\"\nclass DomainSpecificChunkSizes:    \"\"\"    Interview tip: Show you understand domain context    \"\"\"        DOMAINS = {        \"LEGAL\": {            \"size\": \"1000-2000 words\",            \"reason\": \"Sections/clauses are legal units\",            \"example\": \"Contract clause must be complete\",            \"key_metric\": \"Precision > Recall (accuracy critical)\",            \"overlap\": \"20% (preserve clause boundaries)\",        },                \"MEDICAL\": {            \"size\": \"500-1000 words\",            \"reason\": \"Patient outcomes depend on complete context\",            \"example\": \"Symptoms + test results + diagnosis\",            \"key_metric\": \"Recall > Precision (miss nothing)\",            \"overlap\": \"20% (connect symptoms to outcomes)\",        },                \"E-COMMERCE\": {            \"size\": \"400-800 words\",            \"reason\": \"Product info is naturally separated\",            \"example\": \"Product specs, reviews, shipping info\",            \"key_metric\": \"Speed (real-time product search)\",            \"overlap\": \"10% (less critical)\",        },                \"CUSTOMER_SUPPORT\": {            \"size\": \"600-1000 words\",            \"reason\": \"Q&A pairs with explanation\",            \"example\": \"Question + answer + examples\",            \"key_metric\": \"User satisfaction (answers must be complete)\",            \"overlap\": \"15%\",        },                \"TECHNICAL_DOCS\": {            \"size\": \"500-1000 words\",            \"reason\": \"API docs, parameters need to stay together\",            \"example\": \"Function signature + params + examples\",            \"key_metric\": \"Accuracy (wrong example breaks code)\",            \"overlap\": \"15%\",        },                \"NEWS/MEDIA\": {            \"size\": \"400-600 words\",            \"reason\": \"Articles are already well-written units\",            \"example\": \"One news story = one natural unit\",            \"key_metric\": \"Latency (real-time relevance)\",            \"overlap\": \"10%\",        },    }\n```\n\n**Interview Answer Example:**\n\n```\n\"For a medical chatbot vs an e-commerce bot, chunk sizing would be very different:\nMEDICAL (diagnosing symptoms):- Chunk size: 700-1000 words- Why: Symptoms, tests, diagnosis, treatment must be together- Metric: Optimize for RECALL (don't miss anything)- Overlap: 20% (very important - connect related symptoms)\nE-COMMERCE (product recommendation):- Chunk size: 500-800 words  - Why: Products are naturally separate; specs are self-contained- Metric: Optimize for SPEED (<100ms) and cost- Overlap: 10% (less critical)\nThe key difference: Medical prioritizes completeness; e-commerce prioritizes speed.\"\n❌ \"We just use 512-token chunks like everyone else\"   → Shows no independent thinking\n❌ \"Bigger chunks are always better\"   → Ignores retrieval precision vs recall trade-off\n❌ \"We never tested different chunk sizes\"   → Suggests no systematic approach\n❌ \"Chunk size doesn't matter much\"   → Shows ignorance of its impact\n✅ Instead say:   \"We systematically tested 5 different chunk sizes on our    evaluation set, measuring precision, recall, latency, and cost.    Based on the results, 800-word chunks provided the best balance    for our use case.\"\n```\n\n**Answer:**\n\n```\n\"Good question. This is where recursive chunking + parent-child hierarchy becomes critical.\nApproach:1. DETECT boundaries: Identify natural sections, paragraphs, sentences2. RECURSIVE splitting: Try to split at paragraph level first3. FALL BACK gracefully: If paragraph > chunk_size, split by sentences4. PRESERVE context: Use parent chunks to keep full section\nExample: If I want 800-word chunks but a legal clause is 1200 words:- Don't: Break the clause (loses meaning)- Do: Keep clause as one parent chunk      Split into 2-3 child chunks (for embedding)      Retrieve child chunks + expand to full parent (for LLM)\nThis preserves semantic integrity while optimizing for retrieval.\"\n```\n\n**Answer:**\n\n```\n\"Variable-length documents need adaptive chunking:\n1. ANALYZE document length   - If < 2000 words: Use as single chunk   - If 2000-10000 words: Split into 3-5 chunks   - If > 10000 words: Use hierarchical chunking\n2. USE overlap intelligently   - Short docs: 10% overlap (minimal redundancy)   - Long docs: 20% overlap (ensure continuity)   - Large docs: 25% overlap (bridge multiple chunks)\n3. IMPLEMENT dynamic sizing   chunks = []   if doc_length < 2000:       chunks = [entire_doc]   else:       base_size = doc_length / (num_sections * 0.8)       chunks = recursive_split(doc, size=base_size)\nThis ensures consistency while respecting document structure.\"\n```\n\n**Answer:**\n\n```\n\"Mixed content requires special handling:\nSTRATEGY:1. DETECT content type   if is_table:       chunk_size = 1000_chars  (tables are dense)   elif is_code:       chunk_size = 500_chars   (code needs precision)   else:       chunk_size = 2000_chars  (normal text)\n2. KEEP units intact   - Table: Never split a row across chunks   - Code: Never split a function/class   - Text: Split at paragraph boundary\n3. ADD metadata   Each chunk stores:   - content_type: 'text', 'table', 'code'   - importance: 'high', 'normal', 'low'   - structure: original section/subsection\n4. USE in retrieval   Query for code → prioritize code chunks   Query for stats → prioritize table chunks\nThis ensures retrieval quality across mixed content.\"\n```\n\n**Answer:**\n\n```\n\"Production data often differs from dev data. I'd implement:\n1. MONITORING in production   - Track precision/recall on real queries   - Monitor hallucination rate   - Watch for queries that fail   - Collect user feedback\n2. A/B TESTING for changes   Before changing chunk size, run experiment:   - Control: Current chunk size (95% traffic)   - Test: New chunk size (5% traffic)   - Measure: Impact on user satisfaction, metrics\n3. GRADUAL ROLLOUT   - Week 1: Test new size with 5% queries   - Week 2: Expand to 20% if metrics good   - Week 3: Full rollout or rollback\n4. FALLBACK PLAN   If new size performs worse:   - Immediately revert to previous size   - Investigate issue   - Re-test with adjusted size\nThis ensures production stability while allowing optimization.\"\n```\n\nPrint this before your interview! Interviewers love when you reference a framework.\n\n```\nCHUNK SIZE DECISION MATRIX==========================\nSMALL (400-600w) | MEDIUM (800-1200w) | LARGE (1500-2500w)Precision/Recall Trade  | High Precision   | Balanced           | High RecallLatency                 | Fast (<50ms)     | Moderate (50-80ms) | Slow (>80ms)Cost                    | Low              | Medium             | HighSemantic Preservation   | At Risk          | Good               | ExcellentEmbedding Model Load    | Low              | Medium             | HighLLM Context Budget      | Large margin     | Comfortable        | TightBest For                | E-commerce       | Support/FAQ        | Legal/Medical                        | News             | Tech docs          | Research\nYOUR USE CASE:- Domain: _______- Priority: Speed? Cost? Accuracy?- Document length: Short / Medium / Long- Precision vs Recall: Which matters more?\nRECOMMENDED SIZE: _______ words\n```\n\nWhen asked “How do you decide chunk size?”, end with this:\n\n```\n\"In summary, it's a three-step process:\n1. ANALYZE constraints   - Document type (legal vs news vs code)   - Precision vs recall needs   - Latency and cost budgets   - Model capabilities\n2. EMPIRICAL testing   - Test 3-5 different sizes   - Measure precision, recall, latency, cost   - Evaluate hallucination rate   - Select size that balances trade-offs\n3. PRODUCTION monitoring   - Track real-world metrics   - Run A/B tests before changes   - Implement fallback plan   - Iterate based on user feedback\nThe key insight: There's no universal answer. It's a design decision based on YOUR specific constraints and priorities.\nFor this project, I would recommend [size] words because [specific reasons related to their domain].\"\n```\n\n“I’ve researched this topic through:”\n\n[How to Decide Chunk Size in Any Project: Complete Interview Guide](https://pub.towardsai.net/how-to-decide-chunk-size-in-any-project-complete-interview-guide-81558cb49052) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-to-decide-chunk-size-in-any-project-complete-interview-guide", "canonical_source": "https://pub.towardsai.net/how-to-decide-chunk-size-in-any-project-complete-interview-guide-81558cb49052?source=rss----98111c9905da---4", "published_at": "2026-08-12 22:31:01+00:00", "updated_at": "2026-08-12 22:46:45.948277+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools"], "entities": ["OpenAI", "Anthropic", "Cohere", "Claude 3 Sonnet", "text-embedding-3-large"], "alternates": {"html": "https://wpnews.pro/news/how-to-decide-chunk-size-in-any-project-complete-interview-guide", "markdown": "https://wpnews.pro/news/how-to-decide-chunk-size-in-any-project-complete-interview-guide.md", "text": "https://wpnews.pro/news/how-to-decide-chunk-size-in-any-project-complete-interview-guide.txt", "jsonld": "https://wpnews.pro/news/how-to-decide-chunk-size-in-any-project-complete-interview-guide.jsonld"}}