RAG Can Retrieve the Right Document and Still Hide the Answer

A developer demonstrated with a self-contained SQLite FTS5 example that a RAG pipeline can retrieve the correct document ID while the text passed to the model omits the answer, because a fixed character prefix and a query-centered snippet are different views of the same source. The constructed test, which needs no language model or API key, shows the return-window fact present in the full document and in the snippet but absent from the first 180 characters, arguing that retrieval checks should be split into candidate retrieval, evidence delivery and answering. The author recommends inspecting the actual assembled model input after extraction, truncation and deduplication rather than the full document on disk.

A document ID can pass a retrieval check while the text sent to the model omits the answer. That leaves a gap between two questions that often get merged during RAG debugging: Here is a small, constructed example that makes the gap visible without calling a language model. It uses Python's sqlite3 module and a SQLite build with FTS5 enabled. No API key or additional Python package is needed. The fictional question is: What is the return window for Zephyr orders? The source contains the answer, but places it after a long introduction. The query intentionally matches only the relevant document, so this example tests text delivery rather than ranking quality. Save this as evidence window.py and run python3 evidence window.py : python import sqlite3 db = sqlite3.connect ":memory:" db.execute "CREATE VIRTUAL TABLE docs USING fts5 doc id UNINDEXED, body " policy = "General shipping guidance. " 80 + "For Zephyr orders, the return window is 17 days after delivery." db.executemany "INSERT INTO docs VALUES ?, ? ", "returns", policy , "shipping", "Standard shipping takes three to five business days." , , doc id, full text, excerpt = db.execute """ SELECT doc id, body, snippet docs, 1, ' ', ' ', ' ... ', 32 FROM docs WHERE docs MATCH ? ORDER BY bm25 docs LIMIT 1 """, "Zephyr", , .fetchone head = full text :180 marker = "17 days" Known only because this is a constructed example. print "Retrieved document:", doc id print "Marker in full source:", marker in full text print "Marker in first 180 characters:", marker in head print "Marker in query-centered snippet:", marker in excerpt print "Snippet:", excerpt db.close Observed output when run locally with Python 3.12: Retrieved document: returns Marker in full source: True Marker in first 180 characters: False Marker in query-centered snippet: True Snippet: ... For Zephyr orders, the return window is 17 days after delivery. The query and retrieved document are identical for both views. One view is the document's first 180 characters. The other is a fragment selected around the query match. If the check only asks whether doc id == "returns" , both look successful. If the generation step receives only head , however, the relevant source does not provide the answer in the supplied context. A model might still guess correctly or answer from prior knowledge. That would not show that retrieval delivered the evidence. In FTS5, ORDER BY bm25 docs sorts the better matches first because its BM25 scores are numerically smaller for stronger matches. The 1 in snippet docs, 1, ... selects the second column, body . The square brackets mark query matches, and the ellipsis marks omitted text. The final 32 limits the fragment using FTS5 tokens, not the tokens of a language model. These behaviors are documented in the SQLite FTS5 reference https://www.sqlite.org/fts5.html . The 180-character prefix and 32-token snippet are different budgets. This is not a controlled efficiency comparison between them. It is an example of information being available in a source but absent from a particular view of that source. Candidate retrieval: Did the returned IDs include the expected sources? This can reveal failures in queries, ranking, filtering, or candidate limits. It does not tell you which passages reached the model. Evidence delivery: Did the assembled model input contain the facts and conditions needed to answer? Inspect the actual text after extraction, truncation, deduplication, and context assembly. Looking at the full document on disk is insufficient if the model received something smaller. Answering: Did the model use that evidence correctly? It can receive a relevant paragraph and still omit an exception, attach a fact to the wrong entity, or cite a source that does not support its claim. These checks need their own observations. A correct document ID does not establish evidence delivery. A delivered fact does not establish correct reasoning. A correct final answer does not prove that the supplied context supported it. The example uses "17 days" in excerpt because the source and answer were constructed for this demonstration. It is a diagnostic marker, not a general correctness evaluator. The same phrase could appear in an expired policy, a quoted customer question, or a sentence saying that a 17-day window does not apply. Conversely, an answer could express the same duration in different words and fail the string check. For a real evaluation, inspect the supporting passage and its conditions, or use annotated evidence spans when available. Keep answer assessment separate from evidence-presence checks. This example puts the answer next to the query term. That makes a query-centered snippet useful here. It does not show that snippets always contain the answer. The answer might be far from the lexical match, spread across several documents, or qualified by a later exception. A system may need another search or a further read. IRCoT https://aclanthology.org/2023.acl-long.557/ studies the broader problem of interleaving retrieval and reasoning for multi-step questions; this SQLite example does not evaluate that approach. When reviewing a failing RAG trace, I would keep the source ID, source version, exact delivered text, and any location information available from the extraction step. If locations are recorded, specify their units: character offsets, byte offsets, and tokenizer positions are not interchangeable. FTS5's snippet call above does not return a character offset. Then check the final assembled input. A useful excerpt can still be dropped or cut again after retrieval. The local script retrieved the expected document, omitted the known marker from a prefix view, and included it in a query-centered snippet. It did not call a model, measure answer accuracy, compare retrievers, or establish how often this failure occurs in a real corpus. Before replacing a retriever because an answer is wrong, check the text that crossed the retrieval-to-generation boundary. The missing information may already be in the selected source. AI disclosure: This article was drafted and edited by AI agents under the author's direction. The illustrative code was executed locally and its output checked. The documents and question are fictional; no model requests or benchmark runs were made for this article.