{"slug": "rag-chunking-tested-what-actually-improves-retrieval", "title": "RAG Chunking, Tested: What Actually Improves Retrieval", "summary": "A developer built a small labelled RAG corpus of three documents, 18 sentences and nine questions to compare fixed-size, recursive and semantic chunking under TF-IDF retrieval, finding recursive chunking reached recall@3 of 1.00 versus 0.89 for fixed-size, while semantic chunking matched that recall with smaller chunks. The writeup also cites a Mayo Clinic study reporting adaptive chunking lifted answer accuracy from 50% to 87% over a fixed-size baseline, and recommends starting with recursive splitting at roughly 512 tokens with 50-100 tokens of overlap before tuning further.", "body_md": "Chunking is the step in a RAG pipeline where long documents are split into small passages before they are embedded. Each passage becomes one retrievable unit, so the chunk boundaries decide exactly what the retriever can and cannot return. Get them wrong and the correct answer sits in your corpus but never reaches the model.\n\n**Key takeaways**\n\nRecursive splitting beats naive fixed-size chunking in almost every published test, because it keeps sentences and paragraphs intact instead of cutting them mid-thought.\n\nOn our hands-on toy corpus, recursive chunking reached recall@3 of 1.00 against 0.89 for fixed-size, while semantic chunking matched that recall with smaller, tighter chunks.\n\nA Mayo Clinic study found adaptive chunking lifted answer accuracy from 50% to 87% compared with a fixed-size baseline (Bioengineering, 2025).\n\nStart with recursive splitting at about 512 tokens and 50 to 100 tokens of overlap, then measure against your own questions before tuning further.\n\nChunking is necessary but not sufficient: hybrid retrieval and reranking usually recover more quality than further chunk-size tuning.\n\nA RAG pipeline has four stages: chunk, embed, retrieve, generate. Three of them get the attention. People benchmark embedding models, argue about vector databases, and tune the generator's prompts for hours. The first stage is decided once at ingest time and then silently shapes everything downstream, because the retriever can only ever return whole chunks. If the answer to a user's question is split across a boundary, or buried in a 2,000-token chunk of unrelated text, no embedding model and no reranker can fully repair that.\n\nMost chunking advice online was never tested. Blog listicles describe five strategies, declare a winner, and show no numbers. This article does two things differently: a hands-on comparison of three strategies on one small labelled corpus, with runnable code, and a check against published benchmarks, with vendor claims clearly separated from independently verified results.\n\n**Fixed-size chunking** cuts the text every N characters or tokens, optionally with overlap. It is dead simple, fast, and completely blind to meaning: boundaries land mid-sentence and occasionally mid-word. Overlap softens this, so an answer straddling a boundary still appears whole in at least one chunk.\n\n**Recursive chunking** tries separators in order, paragraph breaks, line breaks, sentence ends, then spaces, splitting with the finest one that still fits the size limit, and merges small pieces back together. This is the idea behind LangChain's RecursiveCharacterTextSplitter and LlamaIndex's sentence splitters: sentences survive intact, which matters because retrieval scores whole chunks.\n\n**Semantic chunking** embeds each sentence and starts a new chunk wherever similarity between neighbours drops below a threshold. It follows topic shifts instead of character counts, which suits documents that change subject abruptly. The price is an embedding pass over every sentence at ingest, plus a threshold to tune per corpus.\n\nBeyond these three you will meet sliding windows, proposition-level chunking (one atomic fact per chunk), hierarchical approaches like RAPTOR that index summaries of summaries, and late chunking, where the whole document is embedded first and the embeddings are sliced afterwards. The ICLR 2026 SMARTCHUNK paper tested several of these head to head; its numbers appear below.\n\nTo see the mechanism rather than take it on faith, I built a tiny labelled corpus: three short documents, 18 sentences, nine questions with sentence-level gold answers, so every strategy is judged against the same answers regardless of where it cuts. Retrieval is plain TF-IDF cosine similarity. The metric is recall@3: is the answer sentence inside the top three retrieved chunks?\n\nHere are the three chunkers, compact enough to run as one script:\n\n``` python\nimport re\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.metrics.pairwise import cosine_similarity\n\nDOC = \"\"\"Our refund policy covers all paid plans, including monthly and annual billing.\nYou can request a full refund within 30 days of your first payment, no questions asked.\nAfter 30 days, refunds are prorated based on unused service time.\nTo request a refund, email billing@example.com with your invoice number.\nRefund requests are processed within 5 to 7 business days to the original payment method.\nChargebacks filed with your bank will pause your account until the dispute is resolved.\"\"\"\n\ndef fixed_size(text, size=400, overlap=80):\n    chunks, start = [], 0\n    while start = len(seps):\n            return [t]\n        parts, buf = [], \"\"\n        for piece in t.split(seps[i]):\n            cand = buf + seps[i] + piece if buf else piece\n            if len(cand) > size and buf:\n                parts.append(buf)\n                buf = piece\n            else:\n                buf = cand\n        if buf:\n            parts.append(buf)\n        out = []\n        for p in parts:\n            out += split(p, i + 1) if len(p) > size else [p]\n        return out\n    chunks, cur = [], \"\"\n    for p in split(text):\n        if len(cur) + len(p) + 1 = 2:\n            chunks.append(\" \".join(cur))\n            cur = [sents[i]]\n        else:\n            cur.append(sents[i])\n    chunks.append(\" \".join(cur))\n    return chunks\n\nfor name, fn in [(\"fixed\", fixed_size), (\"recursive\", recursive), (\"semantic\", semantic)]:\n    print(f\"--- {name} ---\")\n    for c in fn(DOC)[:2]:\n        print(\"...\" + c.strip()[-60:].replace(\"\\n\", \" \"))\n```\n\nLook at where the first fixed-size chunk ends:\n\n```\n--- fixed ---\n...in 5 to 7 business days to the original payment method. Char\n--- recursive ---\n... within 5 to 7 business days to the original payment method.\n```\n\nThe fixed-size chunker cut mid-word, leaving a stray \"Char\" dangling at the boundary, while the recursive chunker ended cleanly at the sentence boundary. That is the whole mechanism in miniature: arbitrary boundaries slice answers; structural boundaries preserve them.\n\nThe evaluation loop chunks all three documents with each strategy and scores recall@3 plus MRR (mean reciprocal rank, which rewards putting the right chunk first):\n\n```\n# chunkers as defined above\nDOCS = {\"refunds.md\": DOC, \"onboarding.md\": \"\"\"...\"\"\", \"sla.md\": \"\"\"...\"\"\"}\n# GOLD maps each of the nine questions to its answer sentence indices,\n# e.g. \"How many days do I have to get a full refund?\": [1]\n\ndef chunk_all(strategy):\n    fn = {\"fixed\": fixed_size, \"recursive\": recursive, \"semantic\": semantic}[strategy]\n    return [c for doc in DOCS.values() for c in fn(doc)]\n\ndef evaluate(chunks, k=3):\n    vec = TfidfVectorizer().fit_transform(chunks + list(GOLD))\n    sims = cosine_similarity(vec[len(chunks):], vec[:len(chunks)])\n    recalls, mrrs = [], []\n    for qi, gold in enumerate(GOLD.values()):\n        top = sims[qi].argsort()[::-1][:k]\n        recalls.append(any(SENTS[g] in chunks[i] for i in top for g in gold))\n        rank = next((r + 1 for r, i in enumerate(top)\n                     if any(SENTS[g] in chunks[i] for g in gold)), None)\n        mrrs.append(1 / rank if rank else 0)\n    return sum(recalls) / len(recalls), sum(mrrs) / len(mrrs)\n\nprint(\"\\nstrategy    chunks  recall@3   MRR\")\nfor name in (\"fixed\", \"recursive\", \"semantic\"):\n    chunks = chunk_all(name)\n    r, mrr = evaluate(chunks)\n    print(f\"{name:10s}  {len(chunks):3d}     {r:.2f}    {mrr:.2f}\")\n\n# strategy    chunks  recall@3   MRR\n# fixed         6     0.89    0.89\n# recursive     6     1.00    0.94\n# semantic      9     1.00    0.87\n```\n\nRecursive chunking retrieved the answer sentence in the top three for all nine questions, against eight of nine for fixed-size. Semantic chunking matched recursive on recall but produced nine smaller, topic-tight chunks instead of six, with slightly lower MRR because the ranking spread across more candidates. Read this test for what it is: a toy corpus, TF-IDF instead of embeddings, nine questions. It demonstrates the mechanism; it does not prove a universal ranking. For that, we need the published studies.\n\nA Mayo Clinic team ran one of the cleanest comparisons available (Bioengineering, October 2025): four otherwise identical RAG pipelines differing only in chunking strategy, with 30 clinical questions submitted to each and Gemini 1.0 Pro generating the answers. Adaptive chunking reached 87% answer accuracy against 50% for the baseline (p = 0.001), with retrieval precision of 0.50 versus 0.17, recall of 0.88 versus 0.40, and F1 of 0.64 versus 0.24. Proposition and semantic chunking beat the baseline but trailed the adaptive approach. The key point: aligning chunks to logical topic boundaries is a model-agnostic lever, it improved results without changing the language model at all. Source: Bioengineering 2025, mdpi.com/2306-5354/12/11/1194.\n\nA study of a legal RAG pipeline for EU electricity market regulation (Electronics, 2026) crossed three chunking strategies with two embedding models. Sliding window with bge-small-en-v1.5 gave the best rank-sensitive scores, P@10 of 0.6120 and NDCG@10 of 0.6355, while article-level chunking gave the best top-10 recall at 0.4190. Two lessons: the right chunking depends on whether you need precision or coverage, and chunking interacts with the embedding model, so a strategy that wins with one embedder may not win with another. Source: Electronics 2026, mdpi.com/2079-9292/15/4/749.\n\nThe ICLR 2026 SMARTCHUNK paper adds cost to the picture. On its QA set, plain fixed-size chunking at 512 tokens reached retrieval recall of 0.648, late chunking 0.661, RAPTOR 0.714, and the multi-stage Multi-MAL RAG 0.842. But that last jump cost roughly $0.30 per query and over 4 seconds of latency, against $0.006 and about 1.1 seconds for fixed-size. Diminishing returns are real: SMARTCHUNK's own method landed at 0.829 recall for $0.078. Source: ICLR 2026 proceedings.\n\nNow the honest separation. Framework docs recommend recursive splitting at roughly 512 tokens because practitioners converged on it through experience, not because a controlled trial crowned it, so treat any single \"best chunk size\" number from a vendor blog as a starting point. The standard overlap guidance of 10 to 20% is conventional wisdom in the same sense; one 2026 benchmark roundup attributes up to about 14.5% recall improvement to overlap, but that is a single secondary source, so read the figure as indicative rather than verified.\n\n**Recursive splitting at about 512 tokens, with 50 to 100 tokens of overlap.** Best evidence-to-effort ratio of any default: sentences stay whole, it costs nothing extra, and every serious comparison uses it as the baseline to beat.\n\n**Attach metadata to every chunk**, document title, section heading, date, and filter on it before vector search. Metadata filtering is cheaper than any chunking trick and stacks with all of them.\n\n**Use semantic chunking for documents that change topic abruptly**, transcripts, policy bundles, long reports with uneven structure. Budget the extra embedding pass at ingest, and tune the similarity threshold on your own data instead of copying a blog's value.\n\n**After chunking, add hybrid retrieval and a reranker.** BM25 plus vector search, with a cross-encoder reordering the top candidates, usually recovers more accuracy than further chunk-size tuning.\n\nBuild a golden set: 20 to 50 real user questions with known answer passages, the toy test above scaled up to your documents. Then measure recall@k, how often the right passage lands in the top k retrieved chunks. If the answer exists in your corpus but recall@k is low, the fault is chunking or the retriever, not the language model. The RAGAS framework formalises this as context precision and context recall. If recall is healthy but answers are still wrong, investigate generation and prompting instead; no chunking change fixes a model that ignores good context.\n\nThere is no universal optimal chunk size; studies disagree because their corpora and embedding models differ, which suggests the optimum is a property of your data, not a constant. Most public chunking tests still use older embedding models, so the interaction with current-generation embedders is underexplored. Late chunking and agentic chunking look promising in early papers but lack independent replication at production scale. And the toy test in this article used TF-IDF rather than embeddings, with only nine questions, so read it as a mechanism demo, not as statistics.\n\nChunking is the cheapest lever in a RAG pipeline: decided once at ingest, almost free to change, and it sets the ceiling for everything downstream. Start recursive, measure with a golden set, and let the numbers tell you whether fancier strategies earn their keep.\n\n*Originally published on [LogicStack](https://logicstack.org/blog/rag-chunking-tested). Run the code examples right in your browser with our free interactive [Python](https://logicstack.org/python-playground) and [SQL](https://logicstack.org/sql-playground) playgrounds — no setup, no sign-up.*", "url": "https://wpnews.pro/news/rag-chunking-tested-what-actually-improves-retrieval", "canonical_source": "https://dev.to/ali_82f3409ed625e8ba8515b/rag-chunking-tested-what-actually-improves-retrieval-58aj", "published_at": "2026-10-19 04:03:25+00:00", "updated_at": "2026-10-19 04:18:56.601931+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["LangChain", "LlamaIndex", "Mayo Clinic", "SMARTCHUNK", "RAPTOR"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rag-chunking-tested-what-actually-improves-retrieval", "markdown": "https://wpnews.pro/news/rag-chunking-tested-what-actually-improves-retrieval.md", "text": "https://wpnews.pro/news/rag-chunking-tested-what-actually-improves-retrieval.txt", "jsonld": "https://wpnews.pro/news/rag-chunking-tested-what-actually-improves-retrieval.jsonld"}}