{"slug": "rag-can-retrieve-the-right-document-and-still-hide-the-answer", "title": "RAG Can Retrieve the Right Document and Still Hide the Answer", "summary": "A developer demonstrated with a self-contained SQLite FTS5 example that a RAG pipeline can retrieve the correct document ID while the text passed to the model omits the answer, because a fixed character prefix and a query-centered snippet are different views of the same source. The constructed test, which needs no language model or API key, shows the return-window fact present in the full document and in the snippet but absent from the first 180 characters, arguing that retrieval checks should be split into candidate retrieval, evidence delivery and answering. The author recommends inspecting the actual assembled model input after extraction, truncation and deduplication rather than the full document on disk.", "body_md": "A document ID can pass a retrieval check while the text sent to the model omits the answer.\n\nThat leaves a gap between two questions that often get merged during RAG debugging:\n\nHere is a small, constructed example that makes the gap visible without calling a language model. It uses Python's `sqlite3` module and a SQLite build with FTS5 enabled. No API key or additional Python package is needed.\n\nThe fictional question is: **What is the return window for Zephyr orders?**\n\nThe source contains the answer, but places it after a long introduction. The query intentionally matches only the relevant document, so this example tests text delivery rather than ranking quality.\n\nSave this as `evidence_window.py` and run `python3 evidence_window.py`:\n\n``` python\nimport sqlite3\n\ndb = sqlite3.connect(\":memory:\")\ndb.execute(\"CREATE VIRTUAL TABLE docs USING fts5(doc_id UNINDEXED, body)\")\n\npolicy = (\n    \"General shipping guidance. \" * 80\n    + \"For Zephyr orders, the return window is 17 days after delivery.\"\n)\ndb.executemany(\n    \"INSERT INTO docs VALUES (?, ?)\",\n    [\n        (\"returns\", policy),\n        (\"shipping\", \"Standard shipping takes three to five business days.\"),\n    ],\n)\n\ndoc_id, full_text, excerpt = db.execute(\n    \"\"\"\n    SELECT doc_id, body, snippet(docs, 1, '[', ']', ' ... ', 32)\n    FROM docs\n    WHERE docs MATCH ?\n    ORDER BY bm25(docs)\n    LIMIT 1\n    \"\"\",\n    (\"Zephyr\",),\n).fetchone()\n\nhead = full_text[:180]\nmarker = \"17 days\"  # Known only because this is a constructed example.\n\nprint(\"Retrieved document:\", doc_id)\nprint(\"Marker in full source:\", marker in full_text)\nprint(\"Marker in first 180 characters:\", marker in head)\nprint(\"Marker in query-centered snippet:\", marker in excerpt)\nprint(\"Snippet:\", excerpt)\ndb.close()\n```\n\nObserved output when run locally with Python 3.12:\n\n```\nRetrieved document: returns\nMarker in full source: True\nMarker in first 180 characters: False\nMarker in query-centered snippet: True\nSnippet:  ... For [Zephyr] orders, the return window is 17 days after delivery.\n```\n\nThe query and retrieved document are identical for both views. One view is the document's first 180 characters. The other is a fragment selected around the query match.\n\nIf the check only asks whether `doc_id == \"returns\"`, both look successful. If the generation step receives only `head`, however, the relevant source does not provide the answer in the supplied context.\n\nA model might still guess correctly or answer from prior knowledge. That would not show that retrieval delivered the evidence.\n\nIn FTS5, `ORDER BY bm25(docs)` sorts the better matches first because its BM25 scores are numerically smaller for stronger matches.\n\nThe `1` in `snippet(docs, 1, ...)` selects the second column, `body`. The square brackets mark query matches, and the ellipsis marks omitted text. The final `32` limits the fragment using FTS5 tokens, not the tokens of a language model. These behaviors are documented in the [SQLite FTS5 reference](https://www.sqlite.org/fts5.html).\n\nThe 180-character prefix and 32-token snippet are different budgets. This is not a controlled efficiency comparison between them. It is an example of information being available in a source but absent from a particular view of that source.\n\n**Candidate retrieval:** Did the returned IDs include the expected sources? This can reveal failures in queries, ranking, filtering, or candidate limits. It does not tell you which passages reached the model.\n\n**Evidence delivery:** Did the assembled model input contain the facts and conditions needed to answer? Inspect the actual text after extraction, truncation, deduplication, and context assembly. Looking at the full document on disk is insufficient if the model received something smaller.\n\n**Answering:** Did the model use that evidence correctly? It can receive a relevant paragraph and still omit an exception, attach a fact to the wrong entity, or cite a source that does not support its claim.\n\nThese checks need their own observations. A correct document ID does not establish evidence delivery. A delivered fact does not establish correct reasoning. A correct final answer does not prove that the supplied context supported it.\n\nThe example uses `\"17 days\" in excerpt` because the source and answer were constructed for this demonstration. It is a diagnostic marker, not a general correctness evaluator.\n\nThe same phrase could appear in an expired policy, a quoted customer question, or a sentence saying that a 17-day window does **not** apply. Conversely, an answer could express the same duration in different words and fail the string check.\n\nFor a real evaluation, inspect the supporting passage and its conditions, or use annotated evidence spans when available. Keep answer assessment separate from evidence-presence checks.\n\nThis example puts the answer next to the query term. That makes a query-centered snippet useful here. It does not show that snippets always contain the answer.\n\nThe answer might be far from the lexical match, spread across several documents, or qualified by a later exception. A system may need another search or a further read. [IRCoT](https://aclanthology.org/2023.acl-long.557/) studies the broader problem of interleaving retrieval and reasoning for multi-step questions; this SQLite example does not evaluate that approach.\n\nWhen reviewing a failing RAG trace, I would keep the source ID, source version, exact delivered text, and any location information available from the extraction step. If locations are recorded, specify their units: character offsets, byte offsets, and tokenizer positions are not interchangeable. FTS5's `snippet()` call above does not return a character offset.\n\nThen check the final assembled input. A useful excerpt can still be dropped or cut again after retrieval.\n\nThe local script retrieved the expected document, omitted the known marker from a prefix view, and included it in a query-centered snippet.\n\nIt did not call a model, measure answer accuracy, compare retrievers, or establish how often this failure occurs in a real corpus.\n\nBefore replacing a retriever because an answer is wrong, check the text that crossed the retrieval-to-generation boundary. The missing information may already be in the selected source.\n\n*AI disclosure: This article was drafted and edited by AI agents under the author's direction. The illustrative code was executed locally and its output checked. The documents and question are fictional; no model requests or benchmark runs were made for this article.*", "url": "https://wpnews.pro/news/rag-can-retrieve-the-right-document-and-still-hide-the-answer", "canonical_source": "https://dev.to/haoxiangli/rag-can-retrieve-the-right-document-and-still-hide-the-answer-2k2k", "published_at": "2026-10-19 04:41:57+00:00", "updated_at": "2026-10-19 04:48:01.921021+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-search"], "entities": ["SQLite", "FTS5", "BM25"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rag-can-retrieve-the-right-document-and-still-hide-the-answer", "markdown": "https://wpnews.pro/news/rag-can-retrieve-the-right-document-and-still-hide-the-answer.md", "text": "https://wpnews.pro/news/rag-can-retrieve-the-right-document-and-still-hide-the-answer.txt", "jsonld": "https://wpnews.pro/news/rag-can-retrieve-the-right-document-and-still-hide-the-answer.jsonld"}}