Three Papers Say Your Agent's Memory Loses to Plain RAG

Three research groups publishing between 21 and 26 October found that elaborate agent memory systems underperform plain retrieval-augmented generation over raw chat history, with He et al. at UIUC and Tsinghua naming the loss a "memory tax" and testing 11 open-source memory systems across three benchmarks and three backbones. On LongMemEval-s with a Qwen3.6-27B backbone, two-round RAG scored 81.4% accuracy and none of the 11 memory systems beat it, ranging from A-Mem at 79.4% (−2.0) down to OpenViking at 58.8% (−22.6). The authors attribute the gap to facts lost during ingestion, retrieval fragmentation, and fragility when the reader model differs from the one that built the memory, and recommend reporting naive RAG baselines alongside memory results and keeping a retrieval path to raw history.

What gets lost when an agent rewrites its history into memory, and a laptop test of where it happens Between 21 and 26 October, three separate groups posted the same uncomfortable result. Build an elaborate memory for your agent fact extraction, graphs, hierarchies, learned policies , then compare it against plain retrieval over the raw chat history. Plain retrieval often wins. I've built memory layers like this, so I read all three. Then I ran the cleanest test from one of them on my laptop, to see where the loss actually happens. 💡 TL;DR & Key Takeaways: Three recent papers demonstrate that elaborate agent memory systems often underperform plain RAG over raw chat history, a phenomenon attributed to a "memory tax" caused by fact loss during ingestion, retrieval fragmentation, and model fragility. The author validates these findings with a local reproduction and outlines specific scenarios where memory remains valuable, such as when using weak reader models or when the task requires distilled know-how rather than simple recall. The Memory Tax: He et al. identify three sources of performance loss compared to RAG: facts lost during the write/ingestion phase, evidence that survives but becomes harder to retrieve due to fragmentation or rephrasing, and increased fragility when the model used for reading differs from the one used for building the memory. When Memory Wins: Memory systems are still beneficial when the reader model is too weak to process long raw contexts, when the task requires distilled workflows rather than raw recall, or when a single history is queried enough times to amortize the high initial build cost. Practical Recommendations: To mitigate the tax, authors should report naive RAG baselines alongside memory results, maintain a retrieval path to the raw history for reranking, treat the extraction model as a hard accuracy ceiling, and test memory systems with the specific models intended for production use. What does agent memory add over plain RAG? Every memory system does the same three things. It ingests the conversation history and turns it into something: extracted facts, summaries, a graph, tiered stores. It retrieves some of that at question time. And a model answers from what came back. Naive RAG skips the first step. It chunks the raw history, embeds it, and retrieves chunks. That makes it the honest baseline: whatever a memory system scores above it is what the ingestion bought you. Whatever it scores below is what the ingestion cost. Diagram: naive RAG retrieves from the raw history directly. A memory system first rewrites the history into a store, and both the writing step and the retrieval over rewritten units can lose evidence that RAG would have found. He et al. at UIUC and Tsinghua call that cost the memory tax https://www.alphaxiv.org/abs/2610.agent-memory-net-value-tax , and they ran the comparison at scale: 11 open-source memory systems, three benchmarks, three backbones, all sharing the same embedding model, with retrieval budgets aligned so that nobody wins just by reading more tokens. How often does memory actually beat RAG? On LongMemEval-s, never. Here's their Table 1 with the Qwen3.6-27B backbone: | Method | Type | LongMemEval-s accuracy | vs RAG 2 rounds | |---|---|---|---| | RAG, 2 rounds | raw history | 81.4% | — | | RAG, 1 round | raw history | 77.6% | −3.8 | | A-Mem | flat store | 79.4% | −2.0 | | LightMem | hierarchical | 78.8% | −2.6 | | HippoRAG2 | graph | 78.2% | −3.2 | | MemOS | flat store | 76.6% | −4.8 | | PlugMem | graph | 75.8% | −5.6 | | Mem-T | trained policy | 75.0% | −6.4 | | SimpleMem | flat store | 73.6% | −7.8 | | MemoryOS | hierarchical | 73.4% | −8.0 | | Memory-R2 | trained policy | 73.0% | −8.4 | | MAGMA | graph | 68.2% | −13.2 | | OpenViking | hierarchical | 58.8% | −22.6 | None of the eleven beats two-round RAG on that benchmark with any of the three backbones. On the harder PersonaMem-v2 and MINT-Eval a few systems do pull ahead in some cells Mem-T is +3.9 on MINT-Eval with GPT-5.6 Luna, for example . So the honest version of the headline is "on these benchmarks", not "memory is useless". The other two papers land in the same place from different directions. Bao et al.'s RAW-RAG https://arxiv.org/abs/2610.30755 , dense retrieval plus reranking over raw history, ranks first in six of seven benchmark-model settings, 0.13 to 5.53 points above the best agentic memory system. Peng's MemoryTax: The Bitter Lesson of Agent Memory https://arxiv.org/abs/2610.29823 is a small study, one LoCoMo conversation and one run per cell, but it adds cost: Mem0's build alone cost more than a whole full-context run with the same model $25.88 against $17.84 with Claude Opus 5.5 , and both ended at 95%. Where does the memory tax come from? This is the part I found most useful, because it tells you what to check in your own system. He et al. split the tax into three sources. Facts lost at write time. They traced sampled facts from the raw history into each system's store. The share that survived with no loss ranges from 98.4% for MemoryOS down to 39.4% for OpenViking, and 38.0% of OpenViking's sampled facts had no memory unit at all. If the extractor didn't write it down, no retriever will find it. Evidence that's there but harder to reach. Even when the fact survives, memory construction can fragment it across units, dilute it with unrelated content, or rephrase it until it no longer matches how the user asks. They measured how many retrieval rounds it takes to recover all the gold evidence, and memory stores recover fewer queries than raw chunks in the first round. More fragility. Build the memory with one model and read it with another, and many memory systems lose more accuracy than RAG does under the same swap. Long-lived agents change models, so this one bites in production. Peng adds a sharper version of the first point: the model that writes the memory sets a ceiling. On a Zep graph written by GPT-4o-mini, readers from GPT-4o-mini up to Claude Opus 5.5 score 43%, 43% and 45%. A stronger reader can't recover what a weaker writer left out. What happens when I run the test myself? When is building memory still worth it? None of the three papers says never. Read together, they point at three cases: - The reader is weak. In Peng's study, Mem0 beat full context only with GPT-4o-mini 78% against 74% . A model that can't use a long raw context benefits from someone pre-chewing it. - The task needs distilled know-how, not recall. In He et al.'s experiential-memory tests, distilling past traces into workflows beat naive RAG on web tasks and lost on tool-use tasks. - Many questions hit one history. Mem0's cheaper per-question cost would repay its build after 372 to 596 questions on the same history in Peng's numbers. LoCoMo asks 152. The benchmark matters too. He et al. show that a bare IDF-keyword baseline already scores above 50% on some memory benchmarks, and RAG-Hybrid around 70%. When retrieval alone solves most queries, there's little room for memory to add anything, and any loss it introduces shows up clearly. Rules I'm taking from it My reading, not measurements: - Report naive RAG over raw history next to every memory result. Without it, you can't tell what the memory bought. - Keep a path back to the raw history. He et al.'s simplest fix, retrieving from the source alongside memory and reranking both, took SimpleMem on LongMemEval-s from 73.6% to 82.8% and cut its tax from 12.0 to 5.2 points. - Treat the extraction model as part of your accuracy budget. A cheap writer caps every reader after it. - Test memory with the model you'll swap to, not only the one you built with. - Count the build cost per question you actually expect, not per benchmark run. I hit the same shape in a different domain a few weeks ago: on Vietnamese legal retrieval, BM25 beat the RAG frameworks https://zyvop.com/bm25-beat-the-rag-frameworks-on-vietnamese-law-iihod on the retrieval metrics. Two unrelated benchmarks, same lesson. The cheap baseline is a real baseline, and it has to be in the table. All three papers are English-only, and He et al. list that as a limitation. I don't have a Vietnamese long-term-memory benchmark to run this on, so I'm not going to claim anything about it. If you know of one, tell me. How to check this yourself python from future import annotations import argparse import collections import json import os import random import sys import time import types import torch noqa: E402 from transformers import AutoModelForCausalLM, AutoTokenizer noqa: E402 HERE = os.path.dirname os.path.abspath file DATA = os.path.join HERE, "..", "data", "longmemeval oracle.json" RESULTS = os.path.join HERE, "..", "results" TYPES = "single-session-user", "single-session-assistant", "multi-session", "knowledge-update", "temporal-reasoning" BUILD PROMPT = """Below is one past conversation between a user and an AI assistant, dated {date}. Write the memory an assistant should keep from it: a list of short, self-contained facts. Cover facts about the user events, possessions, plans, numbers, names, dates, preferences and any concrete information the assistant gave. Resolve relative dates against the session date when you can. One fact per line, each starting with "- ". No commentary. Conversation: {conv}""" ANSWER PROMPT = """You are a personal assistant with memory of past conversations with this user. Use only the context below to answer. The current date is {qdate}. Context: {ctx} Question: {q} Answer concisely.""" Adapted from the LongMemEval judge prompts Wu et al., 2025 . JUDGE PROMPT = """I will give you a question, a correct answer, and a response from a model. Answer yes if the response contains the correct answer, or is equivalent to it, or contains all the intermediate steps needed to get it. If the response contains only a subset of the required information, answer no.{extra} Question: {q} Correct answer: {a} Model response: {r} Is the model response correct? Answer yes or no only.""" EXTRA = { "temporal-reasoning": " Do not penalize off-by-one errors in counts of days, weeks or months.", "knowledge-update": " If the response contains old information alongside the updated answer, it is still correct as long as the updated answer is what it gives.", } class LM: def init self, model id, device : self.id = model id self.tok = AutoTokenizer.from pretrained model id self.model = AutoModelForCausalLM.from pretrained model id, dtype=torch.bfloat16 .to device .eval self.device = device self.kw = {"enable thinking": False} if "qwen3" in model id.lower else {} @torch.no grad def call self, prompt, max new : text = self.tok.apply chat template {"role": "user", "content": prompt} , tokenize=False, add generation prompt=True, self.kw enc = self.tok text, return tensors="pt" .to self.device out = self.model.generate enc, do sample=False, max new tokens=max new, pad token id=self.tok.pad token id or self.tok.eos token id text = self.tok.decode out 0, enc "input ids" .shape 1 : , skip special tokens=True .strip if self.device == "mps": torch.mps.empty cache return text def device : return "mps" if torch.backends.mps.is available else "cuda" if torch.cuda.is available else "cpu" def pick n, seed : data = json.load open DATA data = x for x in data if x "question type" in TYPES and not x "question id" .endswith " abs" rng = random.Random seed by = collections.defaultdict list for x in data: by x "question type" .append x per = n // len TYPES return x for t in TYPES for x in rng.sample by t , per def conv text session : return "\n".join f"{t 'role' }: {t 'content' }" for t in session def jl read path : if not os.path.exists path : return return json.loads ln for ln in open path def out dir tag : d = os.path.join RESULTS, tag if tag else RESULTS os.makedirs d, exist ok=True return d def cmd build a : path = os.path.join out dir a.tag , "memory units.jsonl" done = {r "question id" for r in jl read path } lm = LM a.builder, device t0 = time.time qs = pick a.n, a.seed with open path, "a" as f: for j, x in enumerate qs : if x "question id" in done: continue units = for date, sess in zip x "haystack dates" , x "haystack sessions" : out = lm BUILD PROMPT.format date=date, conv=conv text sess , a.max new facts = ln 2: .strip for ln in out.splitlines if ln.strip .startswith "- " units.append {"date": date, "facts": facts} f.write json.dumps {"question id": x "question id" , "builder": a.builder, "units": units} + "\n" f.flush print f"build {j + 1}/{len qs } {sum len u 'facts' for u in units } facts {time.time - t0:.0f}s", flush=True def contexts x, mem : gold turns = for date, sess in zip x "haystack dates" , x "haystack sessions" : ts = f" {date} {t 'role' }: {t 'content' }" for t in sess if t.get "has answer" gold turns += ts sessions = "\n\n".join f" Session on {d} \n{conv text s }" for d, s in zip x "haystack dates" , x "haystack sessions" units = "\n".join f" {u 'date' } {fct}" for u in mem "units" for fct in u "facts" return {"gold turns": "\n".join gold turns , "gold sessions": sessions, "memory units": units} def cmd answer a : path = os.path.join out dir a.tag , "answers.jsonl" done = { r "question id" , r "context" for r in jl read path } mem tag = a.tag if a.memory from is None else a.memory from mems = {r "question id" : r for r in jl read os.path.join out dir mem tag , "memory units.jsonl" } qs = x for x in pick a.n, a.seed if x "question id" in mems lm = LM a.answerer, device t0 = time.time with open path, "a" as f: for j, x in enumerate qs : for name, ctx in contexts x, mems x "question id" .items : if x "question id" , name in done: continue r = lm ANSWER PROMPT.format qdate=x "question date" , ctx=ctx, q=x "question" , 128 f.write json.dumps {"question id": x "question id" , "type": x "question type" , "context": name, "answerer": a.answerer, "ctx chars": len ctx , "response": r} + "\n" f.flush print f"answer {j + 1}/{len qs } {time.time - t0:.0f}s", flush=True def cmd judge a : path = os.path.join out dir a.tag , "judged.jsonl" done = { r "question id" , r "context" for r in jl read path } gold = {x "question id" : x for x in json.load open DATA } lm = LM a.judge, device with open path, "a" as f: for r in jl read os.path.join out dir a.tag , "answers.jsonl" : if r "question id" , r "context" in done: continue x = gold r "question id" v = lm JUDGE PROMPT.format extra=EXTRA.get x "question type" , "" , q=x "question" , a=x "answer" , r=r "response" , 4 r = dict r, judge=a.judge, verdict=v, correct=v.lower .startswith "yes" f.write json.dumps r + "\n" f.flush def cmd report a : rows = jl read os.path.join out dir a.tag , "judged.jsonl" acc = collections.defaultdict list for r in rows: acc r "type" , r "context" .append r "correct" acc "ALL", r "context" .append r "correct" ctxs = "gold turns", "gold sessions", "memory units" print "| Question type | n | " + " | ".join ctxs + " | tax turns - units |" print "|---|---|" + "---|" len ctxs + 1 for t in TYPES + "ALL" : cells = acc t, c for c in ctxs if not cells 0 : continue pct = 100 sum c / len c for c in cells print f"| {t} | {len cells 0 } | " + " | ".join f"{p:.1f}%" for p in pct + f" | {pct 0 - pct 2 :+.1f} pp |" per-query overlap, as in the paper's Venn plot by = collections.defaultdict dict for r in rows: by r "question id" r "context" = r "correct" both = sum 1 for v in by.values if v.get "gold turns" and v.get "memory units" only raw = sum 1 for v in by.values if v.get "gold turns" and not v.get "memory units" only mem = sum 1 for v in by.values if not v.get "gold turns" and v.get "memory units" print f"\nPer-query: both {both}, raw-only tax {only raw}, memory-only benefit {only mem}, n={len by }" mem tag = a.tag if a.memory from is None else a.memory from mems = jl read os.path.join out dir mem tag , "memory units.jsonl" nf = sum len u "facts" for u in m "units" for m in mems print f"Memory units per question: mean {sum nf / max len nf , 1 :.1f}" lens = collections.defaultdict list for r in rows: lens r "context" .append r "ctx chars" print "Mean context chars: " + ", ".join f"{c} {sum v / len v :.0f}" for c, v in lens.items def main : ap = argparse.ArgumentParser ap.add argument "cmd", choices= "build", "answer", "judge", "report" ap.add argument "--n", type=int, default=60 ap.add argument "--seed", type=int, default=0 ap.add argument "--builder", default="Qwen/Qwen3-1.7B" ap.add argument "--answerer", default="Qwen/Qwen3-1.7B" ap.add argument "--judge", default="Qwen/Qwen3-4B" ap.add argument "--max-new", type=int, default=400 ap.add argument "--tag", default="", help="results subfolder; empty = results/" ap.add argument "--memory-from", default=None, help="tag whose memory units.jsonl to read writer/reader swap ; default = --tag" a = ap.parse args {"build": cmd build, "answer": cmd answer, "judge": cmd judge, "report": cmd report} a.cmd a if name == " main ": main Share your thoughts in the comments — I'd love to hear how this technology is impacting your industry.