{"slug": "why-does-rag-miss-information-that-s-clearly-in-the-document", "title": "Why Does RAG Miss Information That's Clearly in the Document?", "summary": "Rijul, a developer building the AI code review tool LiveReview, explains why retrieval-augmented generation systems fail to surface information that is present in a source document. The core issue, he writes, is chunking: splitting text into pieces that are too large, too small, or that cut across ideas limits what retrieval can hand to the model, capping answer accuracy. He outlines common strategies including fixed-size, overlapping, structure-aware, semantic, and parent-child chunking to mitigate the problem.", "body_md": "*Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nYou have the information in your document. You know it's there.\n\nBut when you ask your RAG system about it, the model gives an answer that misses it completely.\n\nWhy?\n\nIn RAG, the model can only answer using the pieces of text that are retrieved.\n\nAnd those pieces of text depend heavily on how the document was split into chunks.\n\nIf the information is split badly, it might never reach the model in the right form, even though it is clearly present in the original document.\n\nLet's first go through a quick overview of how RAG works.\n\nThe LLM does not necessarily see your whole document. It usually sees only the chunks that are retrieved.\n\n**Chunking determines what information can be retrieved and shown to the model.**\n\nIf the information you need is not present in the retrieved chunks, the model cannot use it.\n\nThis means retrieval puts a ceiling on how accurate the answer can be. Even if the LLM is capable of answering the question, it cannot recover information that was never provided to it.\n\nSuppose you have a large chunk that covers five different topics.\n\nWhen this chunk is converted into an embedding, its representation captures information about all of those topics.\n\nNow imagine someone asks about just one specific topic.\n\nThe chunk may still be retrieved, but its representation is not focused only on that topic. The other topics can make the match less precise.\n\nSo, very large chunks can make retrieval less focused.\n\nNow imagine that a chunk contains only a single sentence.\n\nThat sentence might depend on the sentences before it to make sense.\n\nWithout that surrounding context, the chunk may not contain enough information to understand what it is talking about.\n\nSo, making chunks too small can also hurt retrieval.\n\nSuppose a paragraph contains one complete idea, but you split it right in the middle.\n\nNow the two chunks contain only parts of that idea.\n\nIf the relevant information is split across chunks, retrieving only one of them may not give the LLM enough context to understand the full idea.\n\nThis is why simply cutting text into arbitrary pieces can cause problems.\n\nBecause of these issues, there are different ways to decide where one chunk should end and another should begin.\n\nLet's look at some common strategies.\n\nThis splits the document into chunks containing a fixed number of tokens.\n\nFor example, you might create a new chunk every 500 tokens.\n\nIt is simple, but it does not care about the meaning of the text. It can cut a sentence, paragraph, or idea in the middle.\n\nThis is similar to fixed-size chunking, but consecutive chunks share some text.\n\nFor example:\n\n```\nChunk 1: A B C D E F\nChunk 2:       E F G H I J\n```\n\nThe overlap helps preserve some context when an idea happens to cross a chunk boundary.\n\nInstead of immediately cutting at an arbitrary token count, the system tries to preserve the document's structure.\n\nIt might first split by headings, then paragraphs, then sentences, using smaller units only when necessary.\n\nThis helps keep related content together.\n\nSemantic chunking tries to identify where the topic or meaning changes.\n\nInstead of asking only:\n\n\"Have we reached 500 tokens?\"\n\nit asks something closer to:\n\n\"Has the topic changed enough that this should become a new chunk?\"\n\nThis can produce more meaningful chunks, but it generally requires additional processing.\n\nHere, retrieval happens using smaller **child chunks**, but when a relevant child is found, the system can provide the larger **parent section** to the LLM.\n\nThis gives retrieval a focused unit to search while still providing more surrounding context to the model.\n\nHere, additional context is added to each chunk to explain where it came from or what it represents.\n\nFor example, a chunk might be accompanied by information about its document, section, or surrounding context.\n\nThis can help the retrieval system and the LLM interpret the chunk more accurately.\n\nThere is no single way to split a document into chunks.\n\nThe goal is to create chunks that are **focused enough to retrieve accurately, while containing enough context to preserve their meaning**.\n\nIf the chunks are too large, retrieval can become less focused.\n\nIf they are too small, important context can be lost.\n\nAnd if the boundaries break apart ideas, the LLM may never receive the information it needs in a usable form.\n\nSo before blaming the LLM for a bad RAG answer, it is worth looking at something much earlier in the pipeline:\n\n**How did you split the document in the first place?**\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n⭐ Star it on GitHub: \n\nLiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.\n\n*LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.*\n\n| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | \n|---|---|---|\n\n**Here's the goal:**\n\n**Click below to try LiveReview with your codebase:**", "url": "https://wpnews.pro/news/why-does-rag-miss-information-that-s-clearly-in-the-document", "canonical_source": "https://dev.to/rijultp/why-does-rag-miss-information-thats-clearly-in-the-document-2plk", "published_at": "2026-09-21 18:04:28+00:00", "updated_at": "2026-09-21 18:31:32.404115+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-tools", "developer-tools"], "entities": ["Rijul", "LiveReview", "HexmosTech"], "alternates": {"html": "https://wpnews.pro/news/why-does-rag-miss-information-that-s-clearly-in-the-document", "markdown": "https://wpnews.pro/news/why-does-rag-miss-information-that-s-clearly-in-the-document.md", "text": "https://wpnews.pro/news/why-does-rag-miss-information-that-s-clearly-in-the-document.txt", "jsonld": "https://wpnews.pro/news/why-does-rag-miss-information-that-s-clearly-in-the-document.jsonld"}}