# Why Does RAG Miss Information That's Clearly in the Document?

> Source: <https://dev.to/rijultp/why-does-rag-miss-information-thats-clearly-in-the-document-2plk>
> Published: 2026-09-21 18:04:28+00:00

*Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.*

You have the information in your document. You know it's there.

But when you ask your RAG system about it, the model gives an answer that misses it completely.

Why?

In RAG, the model can only answer using the pieces of text that are retrieved.

And those pieces of text depend heavily on how the document was split into chunks.

If the information is split badly, it might never reach the model in the right form, even though it is clearly present in the original document.

Let's first go through a quick overview of how RAG works.

The LLM does not necessarily see your whole document. It usually sees only the chunks that are retrieved.

**Chunking determines what information can be retrieved and shown to the model.**

If the information you need is not present in the retrieved chunks, the model cannot use it.

This means retrieval puts a ceiling on how accurate the answer can be. Even if the LLM is capable of answering the question, it cannot recover information that was never provided to it.

Suppose you have a large chunk that covers five different topics.

When this chunk is converted into an embedding, its representation captures information about all of those topics.

Now imagine someone asks about just one specific topic.

The chunk may still be retrieved, but its representation is not focused only on that topic. The other topics can make the match less precise.

So, very large chunks can make retrieval less focused.

Now imagine that a chunk contains only a single sentence.

That sentence might depend on the sentences before it to make sense.

Without that surrounding context, the chunk may not contain enough information to understand what it is talking about.

So, making chunks too small can also hurt retrieval.

Suppose a paragraph contains one complete idea, but you split it right in the middle.

Now the two chunks contain only parts of that idea.

If the relevant information is split across chunks, retrieving only one of them may not give the LLM enough context to understand the full idea.

This is why simply cutting text into arbitrary pieces can cause problems.

Because of these issues, there are different ways to decide where one chunk should end and another should begin.

Let's look at some common strategies.

This splits the document into chunks containing a fixed number of tokens.

For example, you might create a new chunk every 500 tokens.

It is simple, but it does not care about the meaning of the text. It can cut a sentence, paragraph, or idea in the middle.

This is similar to fixed-size chunking, but consecutive chunks share some text.

For example:

```
Chunk 1: A B C D E F
Chunk 2:       E F G H I J
```

The overlap helps preserve some context when an idea happens to cross a chunk boundary.

Instead of immediately cutting at an arbitrary token count, the system tries to preserve the document's structure.

It might first split by headings, then paragraphs, then sentences, using smaller units only when necessary.

This helps keep related content together.

Semantic chunking tries to identify where the topic or meaning changes.

Instead of asking only:

"Have we reached 500 tokens?"

it asks something closer to:

"Has the topic changed enough that this should become a new chunk?"

This can produce more meaningful chunks, but it generally requires additional processing.

Here, retrieval happens using smaller **child chunks**, but when a relevant child is found, the system can provide the larger **parent section** to the LLM.

This gives retrieval a focused unit to search while still providing more surrounding context to the model.

Here, additional context is added to each chunk to explain where it came from or what it represents.

For example, a chunk might be accompanied by information about its document, section, or surrounding context.

This can help the retrieval system and the LLM interpret the chunk more accurately.

There is no single way to split a document into chunks.

The goal is to create chunks that are **focused enough to retrieve accurately, while containing enough context to preserve their meaning**.

If the chunks are too large, retrieval can become less focused.

If they are too small, important context can be lost.

And if the boundaries break apart ideas, the LLM may never receive the information it needs in a usable form.

So before blaming the LLM for a bad RAG answer, it is worth looking at something much earlier in the pipeline:

**How did you split the document in the first place?**

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub: 

LiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.

*LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.*

| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | 
|---|---|---|

**Here's the goal:**

**Click below to try LiveReview with your codebase:**