# Your RAG System Is Lying To You About That Table > Source: > Published: 2026-07-07 02:17:32+00:00 Retrieval Augmented Generation, or RAG, has become the default way to ask questions about long documents. You do not train a model on your data. You just fetch the right pieces of text and hand them to the model at query time. It works well for plain text. It gets much harder when your document is full of tables, and recent work in the field is starting to admit this out loud. A 2026 benchmark called REAL-MM-RAG built specifically around table heavy documents found that current retrieval models have real trouble understanding tables well, even when the surrounding text is handled fine. That gap is not a rare bug. It shows up constantly in real systems once tables get long or messy. I want to talk about one specific case. Think about a long report full of tables. Some tables run across two or three pages. The header row is on page one. The numbers continue on page two. And often, the real explanation of what a number means sits far away from the table. It might be a small note at the bottom of the page. It might be a sentence in a paragraph three pages later. A human reader connects these pieces without effort. A RAG system often cannot. **The chunking problem** Most RAG systems split documents into chunks before they search. Chunking is usually based on character count or paragraph breaks. This works fine for prose. It breaks badly for tables. If a table spans two pages, a naive chunker may cut it right in the middle. Now you have one chunk with the header row and no data. And another chunk with data rows and no header. If the system retrieves only one of these chunks, the model sees numbers with no idea what they mean. This exact failure shows up often enough that people are now writing evaluation tests just for it. One recent guide on RAG reliability argues that table heavy systems need their own test set, one that checks whether a retrieved chunk still carries its column labels and whether a cited row can actually support the answer being given. **The missing context problem** Now add the footnote issue. Say a table shows a number for revenue. But that number only makes sense with the note below it that says the number excludes one time charges. If the chunk with the table does not include that footnote, the model may state the number as if it is the full, clean value. That is not a small mistake. In many domains, a missing footnote can completely change how a number should be read. The same thing happens with descriptions placed in the surrounding text. Someone might explain a table two paragraphs after it appears, or even on a different page. Standard retrieval treats these as separate pieces of text. It may fetch one and miss the other. There is no built in sense that they belong together. **The on demand problem** There is another layer to this. Some systems do not keep a permanent index of the document. Instead, they read and search the document fresh at query time. This is useful when documents change often or when you do not want to store data long term. But it also means there is no chance to pre process the document carefully in advance. Every table, every footnote, every cross reference has to be found and connected in the moment, quickly, with no earlier pass to fix the structure. This makes the table and footnote problem even harder. You are asking the system to both understand the layout and answer the question, all at once, without a prior clean up step. **Why this matters** None of this means RAG is broken. It means plain text RAG was never built with complex tables in mind. Tables carry meaning through position, not just words. A cell means something because of its row and its column. Text chunking does not know about rows and columns. It only knows about characters and breaks. I do not have a full fix to offer here. Better table aware chunking helps. Keeping headers attached to every piece of a split table helps. Linking footnotes to the exact cells they explain helps. But these are still open problems, especially when you are searching a document at query time instead of working from a document you have already indexed and cleaned. Some researchers are trying a different route entirely. A recent paper from a team at Yellow.ai proposes reading documents in page batches using a model that can see the layout, not just the words, so a table that continues past one batch does not lose its meaning at the seam. It is an early idea, but it shows the field agrees this is a real gap, not just a small annoyance. I think this is worth more attention than it gets. A lot of real world documents are not clean paragraphs. They are dense tables, spread across pages, with meaning hiding in the small print. If we want RAG to work well on these documents, we need to think past chunking text and start thinking about chunking structure. I would love to hear if others have found good ways to handle this. Sources REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark