{"slug": "rag-s-trust-problem-citations-memory-and-benchmarks-don-t-add-up-quite-right", "title": "RAG's Trust Problem: Citations, Memory, and Benchmarks Don't Add Up Quite Right", "summary": "A weekly RAG/GraphRAG research digest highlights new work on retrieval-augmented generation's trust and memory problems, including a paper finding that 1.2%–15.7% of RAG citations across model families trace to parametric memory rather than retrieved context. Other papers introduce MIRIX-2.0, a multi-agent memory system claiming state of the art on LoCoMo and LongMemEval, and MAPE, which evolves memory-management programs for up to 12% absolute accuracy gains at a third of the serving cost. A TREC RAG 2026 system paper reports that writer-model choice and adjacent-chunk expansion mattered more than retriever choice.", "body_md": "This digest covers RAG/GraphRAG developments from 2026-11-09 through 2026-11-16: a dense week on arXiv and Hugging Face Daily Papers, a seven-post knowledge-layer push from Neo4j, and one methodology post each from LangChain and LlamaIndex.\n\n[A 45-Minute Answer to One Question](https://neo4j.com/blog/genai/a-45-minute-answer-to-one-question/) — graph-grounded context turns hours of log-diving into one query.\n\n[Cited but Unused: Parametric Provenance in Retrieval-Augmented Generation](https://arxiv.org/abs/2611.17161) — up to 15.7% of RAG citations don't actually trace to retrieval.\n\n[MIRIX-2.0: Multi-Agent Memory System](https://arxiv.org/abs/2611.12068) — new state of the art on LoCoMo and LongMemEval agent-memory benchmarks.\n\n[Introducing ParseBench-2](https://www.llamaindex.ai/blog/introducing-parsebench-v2) — a severity-weighted benchmark exposes where document parsers break RAG upstream.\n\n[DISCERN: Preserve at Write, Resolve at Read for Persistent LLM-Agent Memory](https://arxiv.org/abs/2611.15729) — argues agent memory's real bottleneck is writing, not retrieving.\n\n**[RAG-FGO: Dynamic Optimization of Query-Rewriting Agents for Generative RAG](https://arxiv.org/abs/2611.17788)** — 2026-11-13 — Jointly optimizes discrete prompt templates and continuous decoding parameters for the query-rewriting stage of RAG instead of tuning them separately and statically. Query rewriting is usually the brittle, hand-tuned piece of production RAG pipelines, so a joint optimization approach is directly applicable to teams fighting noisy retrieval feedback on complex queries.\n\n**[Discovering Effective Agent Memory Designs through Program Evolution](https://arxiv.org/abs/2611.17068)** — 2026-11-12 — Introduces MAPE, which evolves whole memory-management programs (not just hyperparameters) via an iterative diagnosis-proposal loop, gaining up to 12% absolute accuracy over hand-designed baselines on LoCoMo/LongMemEval at a third of the serving cost. Suggests automating memory-architecture search rather than hand-crafting write/read/consolidation policies.\n\n**[RMIT-ADM+S at TREC RAG 2026: Research Agents, Retriever Ablations, and Open-Weight Replication](https://arxiv.org/abs/2611.16425)** — 2026-11-12 — A TREC 2026 RAG track system paper where six of seven submitted runs beat the track median; writer-model choice and adjacent-chunk expansion turned out to be bigger quality levers than retriever choice alone. Useful competitive-benchmark evidence on where to actually spend engineering effort in a RAG pipeline.\n\n**[AutoOntic: Automatic Ontology Induction from Text for Knowledge Graph Construction](https://arxiv.org/abs/2611.15467)** — 2026-11-11 — Fixes a core GraphRAG weakness: schema-free LLM extraction produces flat graphs with no type hierarchy or domain/range constraints. AutoOntic canonicalizes types and relations and mines hierarchy/constraints automatically — a usable post-processing stage to upgrade flat extracted graphs before GraphRAG retrieval.\n\n**[Cited but Unused: Parametric Provenance in Retrieval-Augmented Generation](https://arxiv.org/abs/2611.17161)** — 2026-11-12 — Uses counterfactual substitution (swap a retrieved fact for a false variant and re-query) to measure how often RAG citations actually trace to parametric memory rather than retrieved context: 1.2%–15.7% of citable claims across model families. A hard number on a faithfulness problem that citation-format checks alone won't catch.\n\n**[HiddenPathQA: A Benchmark for Knowledge Graph Question Answering When Questions Hide Their Paths](https://arxiv.org/abs/2611.11802)** — 2026-11-09 — Shows most KGQA benchmarks leak the gold reasoning path in the question's surface tokens, unlike real multi-hop questions that compress chains into a phrase naming no relation. Important for honestly evaluating GraphRAG/KGQA systems — strong scores on existing benchmarks may just reflect surface pattern-matching.\n\n**[DISCERN: Preserve at Write, Resolve at Read for Persistent LLM-Agent Memory](https://arxiv.org/abs/2611.15729)** — 2026-11-11 — Challenges the \"memory is just a retrieval problem\" assumption: on their ENT-310 benchmark, a vanilla RAG reader already matches strong memory systems on recall since retention/currency are near-saturated. Argues the real bottleneck is write-time contradiction/entity-collision handling — a useful reframing for teams assuming better retrieval alone fixes memory.\n\n**[MemCredit: From Relevance to Utility for Decision-Time Memory in Long-Horizon Web Agents](https://arxiv.org/abs/2611.15600)** — 2026-11-11 — Notes that the memories most semantically relevant to an agent's state aren't necessarily those that most improve its next action, proposing selection by estimated decision utility instead of embedding-similarity relevance. A concrete, actionable idea: rank agent memory by downstream action impact, not query-memory similarity.\n\n**[OpticalRAG: Pixel-Space Compression for Token-Efficient Retrieval-Augmented Generation](https://arxiv.org/abs/2611.13675)** — 2026-11-10 — Argues RAG's context-length bottleneck is a modality problem, not an information-density one: rendering a document page as an image and using a vision encoder preserves far more detail per token than text compression. A concrete compression-fidelity tradeoff alternative to summarization for teams hitting context-window costs.\n\n**[Correct Answers from Insufficient Histories: A Controlled Study of Agent Memory Evaluation](https://arxiv.org/abs/2611.11840)** — 2026-11-09 — Shows an agent can match a reference answer even when its visible memory history doesn't actually determine it (a lucky guess), and designs a controlled protocol with byte-identical contexts but different correct answers to detect this. A methodological warning: matching the reference answer is not proof memory was sufficient or used correctly.\n\n**[MIRIX-2.0: Multi-Agent Memory System](https://arxiv.org/abs/2611.12068)** — 2026-11-09 — A multi-agent memory architecture reporting 91.4% on LoCoMo and 72.6% on LongMemEval-S, +14.3 points over the strongest baseline, leading on 4 of 5 QA memory benchmarks. Also surfaced on Hugging Face's Nov 16 Daily Papers listing. A strong reference point for benchmarking agent-memory accuracy/cost tradeoffs.\n\n**[Edge-Native Multimodal RAG: Hardware-Aware Quantization and Architectural Optimization for Consumer GPUs](https://arxiv.org/abs/2611.14924)** — 2026-11-11 — An engineering study of multimodal RAG on consumer GPUs (e.g., Tesla T4), quantifying VRAM tradeoffs between quantized weights and co-located vector indices/KV caches. Directly useful for self-hosting RAG on cheaper hardware instead of cloud GPUs.\n\n**[HiVe-Mem: Interleaving Horizontal Memory Expansion and Vertical Evidence Composition for Multimodal Agents](https://huggingface.co/papers/2611.12872)** — 2026-11-09 — A multimodal agent memory framework combining \"horizontal\" memory expansion with \"vertical\" evidence composition, targeting the failure mode where semantic retrieval over compressed memories misses relevant evidence or can't match heterogeneous query needs. A useful design pattern for multimodal memory/RAG systems over long interaction histories.\n\n**[Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents](https://huggingface.co/papers/2611.14320)** — 2026-11-10 — Identifies \"memory-induced degradation,\" where agents do worse with memory than without, because distilled lessons lose the trajectory evidence that justified them, causing misapplication. The fix — linking each stored lesson back to its originating evidence — is a concrete provenance-tracking pattern for episodic agent memory.\n\n**[MEM-SQL: Memory-Empowered Text-to-SQL](https://huggingface.co/papers/2611.18338)** — 2026-11-13 — Points out that Text-to-SQL benchmarks evaluate questions independently, ignoring session reuse of joins/calculations from earlier queries, and introduces HARE, a history-aware benchmark (1,600 requests, 823 verified queries) to measure this. Relevant for memory-augmented structured-query agents, where \"memory\" means reusable query fragments rather than text passages.\n\nNothing relevant this period — the one post in the window (\"Building strong safety evaluations,\" 2026-11-13) is about general safety-evaluation methodology, not retrieval or RAG evaluation.\n\n**[Meet Relate by Neo4j: An Embedded Database for AI Agents](https://neo4j.com/blog/news/meet-relate-by-neo4j-an-embedded-database-for-ai-agents/)** — 2026-11-11 — An open-source, embedded graph database written in Rust (libraries for Python, JS, Rust, Java, Go, Swift, plus a CLI and WASM build) with full Cypher/GQL support and no server required. A single Cypher query can combine graph traversal with vector and full-text search in one statement, framing agent memory as connected facts rather than loose text blobs — a concrete alternative to stitching together separate vector and graph stores.\n\n**[A 45-Minute Answer to One Question](https://neo4j.com/blog/genai/a-45-minute-answer-to-one-question/)** — 2026-11-13 — A case study on Adyen's \"Architect\" system, which replaces ad hoc incident investigation with three Neo4j graphs — service, code, and infrastructure — queried jointly through a composite database. A precomputed, verified graph lets an agent answer a dependency question with one Cypher query instead of reading thousands of lines of logs, cutting a 45-minute manual investigation to seconds.\n\n**[Build a Knowledge Layer: Give Your AI Agents the Shared Context to Succeed](https://neo4j.com/blog/agentic-ai/build-a-knowledge-layer-give-your-ai-agents-the-shared-context-to-succeed/)** — 2026-11-10 — Opening post of a series defining a \"knowledge layer\" as three components: ontologies, data (virtual graphs connecting the ontology to live systems without migration), and memory. It's intro-level, but it's the conceptual frame the rest of the series — and GraphRAG system design generally — builds on.\n\n**[Build a Knowledge Layer: Map Your Data with an Ontology](https://neo4j.com/blog/graph-database/build-a-knowledge-layer-map-your-data-with-an-ontology/)** — 2026-11-10 — Describes a two-level ontology-as-a-graph pattern where a generic `RELATES` edge stores the real relationship name as a property, plus agent tools (`get_schema_help`, `ontology_guided_query`) that let an agent discover schema instead of baking every label into the system prompt. A reusable, maintainable schema-discovery layer that keeps GraphRAG ontologies from going stale or bloating prompts.\n\n**[Build a Knowledge Layer: Turn Your Existing Data into Context](https://neo4j.com/blog/graph-database/build-a-knowledge-layer-turn-your-existing-data-into-context/)** — 2026-11-10 — Shows two graph-construction paths: a Virtual Graph that models a graph on top of existing warehouse tables and translates Cypher into SQL in-place, and Document Intelligence, which extracts an entity/relationship graph from unstructured documents via prompts with no code. Draws a clean contrast — plain vector similarity vs. a graph that lets an agent ask exact questions — for structured/semi-structured enterprise data.\n\n**[Build a Knowledge Layer: Connect Your Data Sources with a Composite Database](https://neo4j.com/blog/graph-database/build-a-knowledge-layer-connect-your-data-sources-with-a-composite-database/)** — 2026-11-10 — Details Neo4j's composite database pattern for querying multiple independently-owned graphs through one connection without copying data, plus a \"shadow database\" workaround for entity resolution across graphs that can't hold direct cross-constituent relationships. A useful pattern for multi-source GraphRAG where source systems must stay decoupled but still need to be linked.\n\n**[Build a Knowledge Layer: Ground Decisions in Precedent](https://neo4j.com/blog/agentic-ai/build-a-knowledge-layer-ground-decisions-in-precedent/)** — 2026-11-10 — Implements the \"memory\" piece as a context graph where precedent is defined structurally (\"two decisions are comparable when they fail the same policy rule\") rather than by embedding similarity, explicitly rejected as less predictable. A concrete example of agentic retrieval over an agent's own decision history as a deterministic alternative to vector-based agent memory.\n\nNothing relevant this period — none of the week's posts substantively covered retrieval, RAG, reranking, embeddings, knowledge graphs, or agent memory.\n\nNothing relevant this period — the week's archive covered industry news and interviews with no substantive retrieval/RAG/knowledge-graph/agent-memory content.\n\nNothing relevant this period — the one post in the window covered open-model industry/policy dynamics, not retrieval or memory systems.\n\nThree independent threads keep landing on the same conclusion this week: trust your retrieval and memory systems less than their accuracy numbers suggest. \"Cited but Unused\" finds RAG citations that don't trace to retrieval; \"Correct Answers from Insufficient Histories\" finds agents that get the right answer for the wrong reason; \"DISCERN\" finds that near-saturated recall hides a write-time contradiction problem; and HiddenPathQA finds KGQA benchmarks that leak their own answers. Meanwhile Neo4j's seven-post push and papers like AutoOntic make the opposite-but-complementary case: when you replace embedding similarity with an explicit, queryable graph — an ontology, a decision-precedent graph, a service-dependency graph — you get deterministic, auditable retrieval instead of a probabilistic guess. The week's message for anyone running RAG or GraphRAG in production: measure faithfulness and provenance directly, don't just trust the benchmark score, and reach for structure wherever the domain allows it.\n\nWhat's catching your eye in retrieval and memory research this week? Drop a comment below.", "url": "https://wpnews.pro/news/rag-s-trust-problem-citations-memory-and-benchmarks-don-t-add-up-quite-right", "canonical_source": "https://dev.to/felipe0liveira/rags-trust-problem-citations-memory-and-benchmarks-dont-add-up-quite-right-36ok", "published_at": "2026-11-16 12:08:12+00:00", "updated_at": "2026-11-16 12:21:21.556313+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents", "generative-ai"], "entities": ["Neo4j", "LangChain", "LlamaIndex", "MIRIX-2.0", "MAPE", "AutoOntic", "DISCERN", "TREC"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rag-s-trust-problem-citations-memory-and-benchmarks-don-t-add-up-quite-right", "markdown": "https://wpnews.pro/news/rag-s-trust-problem-citations-memory-and-benchmarks-don-t-add-up-quite-right.md", "text": "https://wpnews.pro/news/rag-s-trust-problem-citations-memory-and-benchmarks-don-t-add-up-quite-right.txt", "jsonld": "https://wpnews.pro/news/rag-s-trust-problem-citations-memory-and-benchmarks-don-t-add-up-quite-right.jsonld"}}