80% of RAG failures trace back to one decision made before the first vector is ever stored. Most teams never look at it.
The Wrong Thing to Fix
Your RAG system is giving bad answers. You swap the LLM for a bigger one. Still bad. You rewrite the prompt. Marginally better. You switch embedding models. Barely moves the needle.
Meanwhile, nobody has looked at how the documents were chunked.
This is the most common failure pattern in production RAG systems in 2026, and it is almost entirely invisible during development. The system produces answers. The answers look reasonable in testing. And then users ask real questions and something is quietly, consistently wrong.
80% of RAG failures trace back to the ingestion and chunking layer, not the LLM. Most teams discover this after spending weeks tuning prompts and swapping models while their retrieval quietly returns the wrong context every third query.
What Chunking Is and Why It Matters So Much
When you build a RAG system, you cannot feed an entire document library into a vector database at once. You break documents into chunks — smaller pieces that get individually embedded and stored. When a query arrives, the system retrieves the most relevant chunks, not the most relevant documents.
This means the chunk is the atomic unit of your retrieval system. Everything depends on whether the right chunk surfaces for the right query.
If the chunk is too large, it contains multiple topics and the embedding becomes diluted — the vector represents a mixture of concepts rather than a single coherent idea. Retrieval suffers because nothing matches anything cleanly.
If the chunk is too small, it lacks the surrounding context that gives it meaning. The chunk surfaces correctly but the LLM cannot generate a useful answer from it because critical context was in the adjacent chunk that did not get retrieved.
If the chunks cut across the wrong boundaries — splitting a table halfway, breaking a paragraph mid-sentence, separating a question from its answer — the retrieved content is technically present but practically useless.
The largest controlled comparison of chunking strategies to date tested 36 methods, 6 domains, 5 embedding models, and 1,080 total configurations (Shaukat et al., arXiv:2603.06976, March 2026). It confirmed that content-aware chunking significantly outperforms naive fixed-length splitting, and the gap is not marginal.
The Default Is Wrong
Most teams start with fixed-size chunking. You pick a token count — say, 512 tokens — and every document gets cut into pieces of exactly that size, with or without overlap. It is easy to implement, it is the default in most frameworks, and it produces reliably mediocre retrieval.
Weaviate's September 2025 guide puts a number on the gap: the wrong chunking approach can open a difference of up to 9% in recall between the best and worst methods on the same corpus, with the same retriever.
9% recall sounds small. In a system answering 10,000 queries per day, a 9% recall gap means 900 queries per day where the LLM was missing information it should have had. Some of those will produce noticeably wrong answers. Most will produce subtly incomplete ones — answers that are close enough to pass casual review but wrong enough to matter when someone acts on them.
The January 2026 systematic analysis on arXiv produced a finding that upends conventional wisdom: chunk overlap, the near-universal default of adding 10% to 20% overlap between adjacent chunks to preserve context, provides no measurable benefit in retrieval quality. Teams are adding complexity and storage costs to their chunking pipelines for a technique that the most rigorous analysis to date found does not help.
The Hierarchy That Actually Works
The chunking approach with the strongest evidence behind it in 2026 is hierarchical chunking — sometimes called parent-child chunking.
The idea is straightforward. Documents are indexed at two levels. Large parent chunks — full sections, full paragraphs — capture context. Small child chunks capture specific claims, facts, or data points. When a query arrives, the system retrieves based on the small child chunks (which match more precisely) but returns the surrounding parent chunk (which provides the context the LLM needs to answer usefully).
NVIDIA's internal testing on university presentation decks found that hierarchical chunking improves answer accuracy from 61% with fixed-size chunks to 89%. That is a 28 percentage point improvement from a chunking decision alone — with the same model, the same embedding, and the same vector database.
A 28 point accuracy improvement is not what teams expect to find in their chunking layer. It is what they find when they finally look.
Re-Ranking: The Second Fix Nobody Uses
Even with good chunking, approximate nearest-neighbor search introduces noise. The retrieval step optimizes for speed and will include semantically adjacent chunks that are not actually relevant to the query. This is a property of vector similarity search — it finds things that are conceptually close, not things that are definitively correct.
Re-ranking addresses this. A cross-encoder re-ranker takes the retrieved chunks and scores them again, more carefully, against the actual query. It acts as a quality filter between retrieval and generation.
Cross-encoder re-ranking boosts precision by 18% to 42% compared to retrieval without re-ranking, according to multiple production evaluations. Re-rankers add 50 to 200ms of latency and compute cost — but they reduce LLM token consumption by passing fewer, more relevant chunks. At scale, the LLM cost savings frequently outweigh the re-ranker cost.
Most RAG systems deployed in 2024 and early 2025 do not have a re-ranking step. It was considered an optional optimization rather than a core component. By 2026, re-ranking has moved from optional to expected in production-grade RAG pipelines. Teams running systems without it are leaving significant accuracy on the table.
The Silent Decay Problem
There is one more dimension to the chunking problem that is rarely discussed: RAG systems degrade over time without changing.
A v1 RAG that scored 90 on launch can easily score 60 a year later without a single line of code changing. The world moves, the system does not.
Embedding models improve. The model you chose at launch is likely not the best available option twelve months later. Upgrading embedding models requires re-chunking and re-indexing everything — which most teams plan to do but few actually execute on schedule.
Source documents change. If your knowledge base is built on documents that get updated — policy documents, product documentation, regulatory filings — but your index is not refreshed at the same cadence, you are answering questions from stale context. The system looks like it is working. It is working from outdated information.
Evaluation coverage drifts. The questions your evaluation set was designed around are not necessarily the questions real users are asking six months after launch. A system optimized for the original test questions but misses the evolved user intent will show good numbers on internal benchmarks and bad results in production.
What Good Retrieval Infrastructure Makes Possible
The chunking decisions, the re-ranking layer, the index refresh cadence — all of these matter, but they all rest on the same foundation: a vector database that retrieves accurately and efficiently at the scale your system actually reaches.
Good chunking on a database with poor recall still misses results. The best re-ranking layer cannot recover from retrieved chunks that do not contain the right information to begin with. The architectural layers depend on each other, and the retrieval infrastructure is the layer everything else sits on.
This is why the retrieval database is not a commodity choice. High recall is not a nice-to-have. It is the baseline requirement that makes everything else in the pipeline work as designed.
The teams that get this right build systems that improve over time — better chunking, better re-ranking, better evaluation, all producing measurably better answers. The teams that get it wrong keep swapping models and rewriting prompts while the actual problem sits quietly in their chunking configuration.
Endee is an open-source vector database (Apache 2.0) that delivers the highest recall of any independently benchmarked database — the retrieval foundation that makes everything else in your RAG pipeline work correctly. Free to start at endee.io.