{"slug": "three-papers-say-your-agent-s-memory-loses-to-plain-rag", "title": "Three Papers Say Your Agent's Memory Loses to Plain RAG", "summary": "Three research groups publishing between 21 and 26 October found that elaborate agent memory systems underperform plain retrieval-augmented generation over raw chat history, with He et al. at UIUC and Tsinghua naming the loss a \"memory tax\" and testing 11 open-source memory systems across three benchmarks and three backbones. On LongMemEval-s with a Qwen3.6-27B backbone, two-round RAG scored 81.4% accuracy and none of the 11 memory systems beat it, ranging from A-Mem at 79.4% (−2.0) down to OpenViking at 58.8% (−22.6). The authors attribute the gap to facts lost during ingestion, retrieval fragmentation, and fragility when the reader model differs from the one that built the memory, and recommend reporting naive RAG baselines alongside memory results and keeping a retrieval path to raw history.", "body_md": "*What gets lost when an agent rewrites its history into memory, and a laptop test of where it happens*\n\nBetween 21 and 26 October, three separate groups posted the same uncomfortable result. Build an elaborate memory for your agent (fact extraction, graphs, hierarchies, learned policies), then compare it against plain retrieval over the raw chat history. Plain retrieval often wins.\n\nI've built memory layers like this, so I read all three. Then I ran the cleanest test from one of them on my laptop, to see where the loss actually happens.\n\n**💡 TL;DR & Key Takeaways:**\n\nThree recent papers demonstrate that elaborate agent memory systems often underperform plain RAG over raw chat history, a phenomenon attributed to a \"memory tax\" caused by fact loss during ingestion, retrieval fragmentation, and model fragility. The author validates these findings with a local reproduction and outlines specific scenarios where memory remains valuable, such as when using weak reader models or when the task requires distilled know-how rather than simple recall.\n\n*   **The Memory Tax:** He et al. identify three sources of performance loss compared to RAG: facts lost during the write/ingestion phase, evidence that survives but becomes harder to retrieve due to fragmentation or rephrasing, and increased fragility when the model used for reading differs from the one used for building the memory.\n\n*   **When Memory Wins:** Memory systems are still beneficial when the reader model is too weak to process long raw contexts, when the task requires distilled workflows rather than raw recall, or when a single history is queried enough times to amortize the high initial build cost.\n\n*   **Practical Recommendations:** To mitigate the tax, authors should report naive RAG baselines alongside memory results, maintain a retrieval path to the raw history for reranking, treat the extraction model as a hard accuracy ceiling, and test memory systems with the specific models intended for production use.\n\n## What does agent memory add over plain RAG?\n\nEvery memory system does the same three things. It **ingests** the conversation history and turns it into something: extracted facts, summaries, a graph, tiered stores. It **retrieves** some of that at question time. And a model **answers** from what came back.\n\nNaive RAG skips the first step. It chunks the raw history, embeds it, and retrieves chunks. That makes it the honest baseline: whatever a memory system scores above it is what the ingestion bought you. Whatever it scores below is what the ingestion cost.\n\n*Diagram: naive RAG retrieves from the raw history directly. A memory system first rewrites the history into a store, and both the writing step and the retrieval over rewritten units can lose evidence that RAG would have found.*\n\nHe et al. at UIUC and Tsinghua call that cost the [memory tax](https://www.alphaxiv.org/abs/2610.agent-memory-net-value-tax), and they ran the comparison at scale: 11 open-source memory systems, three benchmarks, three backbones, all sharing the same embedding model, with retrieval budgets aligned so that nobody wins just by reading more tokens.\n\n## How often does memory actually beat RAG?\n\nOn LongMemEval-s, never. Here's their Table 1 with the Qwen3.6-27B backbone:\n\n| Method | Type | LongMemEval-s accuracy | vs RAG (2 rounds) | \n|---|---|---|---|\n| RAG, 2 rounds | raw history | 81.4% | — | \n| RAG, 1 round | raw history | 77.6% | −3.8 | \n| A-Mem | flat store | 79.4% | −2.0 | \n| LightMem | hierarchical | 78.8% | −2.6 | \n| HippoRAG2 | graph | 78.2% | −3.2 | \n| MemOS | flat store | 76.6% | −4.8 | \n| PlugMem | graph | 75.8% | −5.6 | \n| Mem-T | trained policy | 75.0% | −6.4 | \n| SimpleMem | flat store | 73.6% | −7.8 | \n| MemoryOS | hierarchical | 73.4% | −8.0 | \n| Memory-R2 | trained policy | 73.0% | −8.4 | \n| MAGMA | graph | 68.2% | −13.2 | \n| OpenViking | hierarchical | 58.8% | −22.6 | \n\nNone of the eleven beats two-round RAG on that benchmark with any of the three backbones. On the harder PersonaMem-v2 and MINT-Eval a few systems do pull ahead in some cells (Mem-T is +3.9 on MINT-Eval with GPT-5.6 Luna, for example). So the honest version of the headline is \"on these benchmarks\", not \"memory is useless\".\n\nThe other two papers land in the same place from different directions. Bao et al.'s [RAW-RAG](https://arxiv.org/abs/2610.30755), dense retrieval plus reranking over raw history, ranks first in six of seven benchmark-model settings, 0.13 to 5.53 points above the best agentic memory system. Peng's [MemoryTax: The Bitter Lesson of Agent Memory](https://arxiv.org/abs/2610.29823) is a small study, one LoCoMo conversation and one run per cell, but it adds cost: Mem0's build alone cost more than a whole full-context run with the same model ($25.88 against $17.84 with Claude Opus 5.5), and both ended at 95%.\n\n## Where does the memory tax come from?\n\nThis is the part I found most useful, because it tells you what to check in your own system. He et al. split the tax into three sources.\n\n**Facts lost at write time.** They traced sampled facts from the raw history into each system's store. The share that survived with no loss ranges from 98.4% for MemoryOS down to 39.4% for OpenViking, and 38.0% of OpenViking's sampled facts had no memory unit at all. If the extractor didn't write it down, no retriever will find it.\n\n**Evidence that's there but harder to reach.** Even when the fact survives, memory construction can fragment it across units, dilute it with unrelated content, or rephrase it until it no longer matches how the user asks. They measured how many retrieval rounds it takes to recover all the gold evidence, and memory stores recover fewer queries than raw chunks in the first round.\n\n**More fragility.** Build the memory with one model and read it with another, and many memory systems lose more accuracy than RAG does under the same swap. Long-lived agents change models, so this one bites in production.\n\nPeng adds a sharper version of the first point: the model that writes the memory sets a ceiling. On a Zep graph written by GPT-4o-mini, readers from GPT-4o-mini up to Claude Opus 5.5 score 43%, 43% and 45%. A stronger reader can't recover what a weaker writer left out.\n\n## What happens when I run the test myself?\n\n## When is building memory still worth it?\n\nNone of the three papers says never. Read together, they point at three cases:\n\n- **The reader is weak.** In Peng's study, Mem0 beat full context only with GPT-4o-mini (78% against 74%). A model that can't use a long raw context benefits from someone pre-chewing it.\n- **The task needs distilled know-how, not recall.** In He et al.'s experiential-memory tests, distilling past traces into workflows beat naive RAG on web tasks and lost on tool-use tasks.\n- **Many questions hit one history.** Mem0's cheaper per-question cost would repay its build after 372 to 596 questions on the same history in Peng's numbers. LoCoMo asks 152.\n\nThe benchmark matters too. He et al. show that a bare IDF-keyword baseline already scores above 50% on some memory benchmarks, and RAG-Hybrid around 70%. When retrieval alone solves most queries, there's little room for memory to add anything, and any loss it introduces shows up clearly.\n\n## Rules I'm taking from it\n\nMy reading, not measurements:\n\n- Report naive RAG over raw history next to every memory result. Without it, you can't tell what the memory bought.\n- Keep a path back to the raw history. He et al.'s simplest fix, retrieving from the source alongside memory and reranking both, took SimpleMem on LongMemEval-s from 73.6% to 82.8% and cut its tax from 12.0 to 5.2 points.\n- Treat the extraction model as part of your accuracy budget. A cheap writer caps every reader after it.\n- Test memory with the model you'll swap to, not only the one you built with.\n- Count the build cost per question you actually expect, not per benchmark run.\n\nI hit the same shape in a different domain a few weeks ago: on Vietnamese legal retrieval, [BM25 beat the RAG frameworks](https://zyvop.com/bm25-beat-the-rag-frameworks-on-vietnamese-law-iihod) on the retrieval metrics. Two unrelated benchmarks, same lesson. The cheap baseline is a real baseline, and it has to be in the table.\n\nAll three papers are English-only, and He et al. list that as a limitation. I don't have a Vietnamese long-term-memory benchmark to run this on, so I'm not going to claim anything about it. If you know of one, tell me.\n\n## How to check this yourself\n\n``` python\nfrom __future__ import annotations\n\nimport argparse\nimport collections\nimport json\nimport os\nimport random\nimport sys\nimport time\nimport types\nimport torch  # noqa: E402\nfrom transformers import AutoModelForCausalLM, AutoTokenizer  # noqa: E402\n\nHERE = os.path.dirname(os.path.abspath(__file__))\nDATA = os.path.join(HERE, \"..\", \"data\", \"longmemeval_oracle.json\")\nRESULTS = os.path.join(HERE, \"..\", \"results\")\n\nTYPES = [\"single-session-user\", \"single-session-assistant\", \"multi-session\",\n         \"knowledge-update\", \"temporal-reasoning\"]\n\nBUILD_PROMPT = \"\"\"Below is one past conversation between a user and an AI assistant, dated {date}.\nWrite the memory an assistant should keep from it: a list of short, self-contained facts.\nCover facts about the user (events, possessions, plans, numbers, names, dates, preferences)\nand any concrete information the assistant gave. Resolve relative dates against the session\ndate when you can. One fact per line, each starting with \"- \". No commentary.\n\nConversation:\n{conv}\"\"\"\n\nANSWER_PROMPT = \"\"\"You are a personal assistant with memory of past conversations with this user.\nUse only the context below to answer. The current date is {qdate}.\n\nContext:\n{ctx}\n\nQuestion: {q}\nAnswer concisely.\"\"\"\n\n# Adapted from the LongMemEval judge prompts (Wu et al., 2025).\nJUDGE_PROMPT = \"\"\"I will give you a question, a correct answer, and a response from a model. Answer yes if the response contains the correct answer, or is equivalent to it, or contains all the intermediate steps needed to get it. If the response contains only a subset of the required information, answer no.{extra}\n\nQuestion: {q}\nCorrect answer: {a}\nModel response: {r}\n\nIs the model response correct? Answer yes or no only.\"\"\"\n\nEXTRA = {\n    \"temporal-reasoning\": \" Do not penalize off-by-one errors in counts of days, weeks or months.\",\n    \"knowledge-update\": \" If the response contains old information alongside the updated answer, it is still correct as long as the updated answer is what it gives.\",\n}\n\nclass LM:\n    def __init__(self, model_id, device):\n        self.id = model_id\n        self.tok = AutoTokenizer.from_pretrained(model_id)\n        self.model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to(device).eval()\n        self.device = device\n        self.kw = {\"enable_thinking\": False} if \"qwen3\" in model_id.lower() else {}\n\n    @torch.no_grad()\n    def __call__(self, prompt, max_new):\n        text = self.tok.apply_chat_template([{\"role\": \"user\", \"content\": prompt}], tokenize=False,\n                                            add_generation_prompt=True, **self.kw)\n        enc = self.tok(text, return_tensors=\"pt\").to(self.device)\n        out = self.model.generate(**enc, do_sample=False, max_new_tokens=max_new,\n                                  pad_token_id=self.tok.pad_token_id or self.tok.eos_token_id)\n        text = self.tok.decode(out[0, enc[\"input_ids\"].shape[1]:], skip_special_tokens=True).strip()\n        if self.device == \"mps\":\n            torch.mps.empty_cache()\n        return text\n\ndef device():\n    return \"mps\" if torch.backends.mps.is_available() else (\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\ndef pick(n, seed):\n    data = json.load(open(DATA))\n    data = [x for x in data if x[\"question_type\"] in TYPES and not x[\"question_id\"].endswith(\"_abs\")]\n    rng = random.Random(seed)\n    by = collections.defaultdict(list)\n    for x in data:\n        by[x[\"question_type\"]].append(x)\n    per = n // len(TYPES)\n    return [x for t in TYPES for x in rng.sample(by[t], per)]\n\ndef conv_text(session):\n    return \"\\n\".join(f\"{t['role']}: {t['content']}\" for t in session)\n\ndef jl_read(path):\n    if not os.path.exists(path):\n        return []\n    return [json.loads(ln) for ln in open(path)]\n\ndef out_dir(tag):\n    d = os.path.join(RESULTS, tag) if tag else RESULTS\n    os.makedirs(d, exist_ok=True)\n    return d\n\ndef cmd_build(a):\n    path = os.path.join(out_dir(a.tag), \"memory_units.jsonl\")\n    done = {r[\"question_id\"] for r in jl_read(path)}\n    lm = LM(a.builder, device())\n    t0 = time.time()\n    qs = pick(a.n, a.seed)\n    with open(path, \"a\") as f:\n        for j, x in enumerate(qs):\n            if x[\"question_id\"] in done:\n                continue\n            units = []\n            for date, sess in zip(x[\"haystack_dates\"], x[\"haystack_sessions\"]):\n                out = lm(BUILD_PROMPT.format(date=date, conv=conv_text(sess)), a.max_new)\n                facts = [ln[2:].strip() for ln in out.splitlines() if ln.strip().startswith(\"- \")]\n                units.append({\"date\": date, \"facts\": facts})\n            f.write(json.dumps({\"question_id\": x[\"question_id\"], \"builder\": a.builder, \"units\": units}) + \"\\n\")\n            f.flush()\n            print(f\"build {j + 1}/{len(qs)} {sum(len(u['facts']) for u in units)} facts {time.time() - t0:.0f}s\", flush=True)\n\ndef contexts(x, mem):\n    gold_turns = []\n    for date, sess in zip(x[\"haystack_dates\"], x[\"haystack_sessions\"]):\n        ts = [f\"[{date}] {t['role']}: {t['content']}\" for t in sess if t.get(\"has_answer\")]\n        gold_turns += ts\n    sessions = \"\\n\\n\".join(f\"[Session on {d}]\\n{conv_text(s)}\"\n                           for d, s in zip(x[\"haystack_dates\"], x[\"haystack_sessions\"]))\n    units = \"\\n\".join(f\"[{u['date']}] {fct}\" for u in mem[\"units\"] for fct in u[\"facts\"])\n    return {\"gold_turns\": \"\\n\".join(gold_turns), \"gold_sessions\": sessions, \"memory_units\": units}\n\ndef cmd_answer(a):\n    path = os.path.join(out_dir(a.tag), \"answers.jsonl\")\n    done = {(r[\"question_id\"], r[\"context\"]) for r in jl_read(path)}\n    mem_tag = a.tag if a.memory_from is None else a.memory_from\n    mems = {r[\"question_id\"]: r for r in jl_read(os.path.join(out_dir(mem_tag), \"memory_units.jsonl\"))}\n    qs = [x for x in pick(a.n, a.seed) if x[\"question_id\"] in mems]\n    lm = LM(a.answerer, device())\n    t0 = time.time()\n    with open(path, \"a\") as f:\n        for j, x in enumerate(qs):\n            for name, ctx in contexts(x, mems[x[\"question_id\"]]).items():\n                if (x[\"question_id\"], name) in done:\n                    continue\n                r = lm(ANSWER_PROMPT.format(qdate=x[\"question_date\"], ctx=ctx, q=x[\"question\"]), 128)\n                f.write(json.dumps({\"question_id\": x[\"question_id\"], \"type\": x[\"question_type\"],\n                                    \"context\": name, \"answerer\": a.answerer, \"ctx_chars\": len(ctx),\n                                    \"response\": r}) + \"\\n\")\n                f.flush()\n            print(f\"answer {j + 1}/{len(qs)} {time.time() - t0:.0f}s\", flush=True)\n\ndef cmd_judge(a):\n    path = os.path.join(out_dir(a.tag), \"judged.jsonl\")\n    done = {(r[\"question_id\"], r[\"context\"]) for r in jl_read(path)}\n    gold = {x[\"question_id\"]: x for x in json.load(open(DATA))}\n    lm = LM(a.judge, device())\n    with open(path, \"a\") as f:\n        for r in jl_read(os.path.join(out_dir(a.tag), \"answers.jsonl\")):\n            if (r[\"question_id\"], r[\"context\"]) in done:\n                continue\n            x = gold[r[\"question_id\"]]\n            v = lm(JUDGE_PROMPT.format(extra=EXTRA.get(x[\"question_type\"], \"\"), q=x[\"question\"],\n                                       a=x[\"answer\"], r=r[\"response\"]), 4)\n            r = dict(r, judge=a.judge, verdict=v, correct=v.lower().startswith(\"yes\"))\n            f.write(json.dumps(r) + \"\\n\")\n            f.flush()\n\ndef cmd_report(a):\n    rows = jl_read(os.path.join(out_dir(a.tag), \"judged.jsonl\"))\n    acc = collections.defaultdict(list)\n    for r in rows:\n        acc[(r[\"type\"], r[\"context\"])].append(r[\"correct\"])\n        acc[(\"ALL\", r[\"context\"])].append(r[\"correct\"])\n    ctxs = [\"gold_turns\", \"gold_sessions\", \"memory_units\"]\n    print(\"| Question type | n | \" + \" | \".join(ctxs) + \" | tax (turns - units) |\")\n    print(\"|---|---|\" + \"---|\" * (len(ctxs) + 1))\n    for t in TYPES + [\"ALL\"]:\n        cells = [acc[(t, c)] for c in ctxs]\n        if not cells[0]:\n            continue\n        pct = [100 * sum(c) / len(c) for c in cells]\n        print(f\"| {t} | {len(cells[0])} | \" + \" | \".join(f\"{p:.1f}%\" for p in pct) +\n              f\" | {pct[0] - pct[2]:+.1f} pp |\")\n    # per-query overlap, as in the paper's Venn plot\n    by = collections.defaultdict(dict)\n    for r in rows:\n        by[r[\"question_id\"]][r[\"context\"]] = r[\"correct\"]\n    both = sum(1 for v in by.values() if v.get(\"gold_turns\") and v.get(\"memory_units\"))\n    only_raw = sum(1 for v in by.values() if v.get(\"gold_turns\") and not v.get(\"memory_units\"))\n    only_mem = sum(1 for v in by.values() if not v.get(\"gold_turns\") and v.get(\"memory_units\"))\n    print(f\"\\nPer-query: both {both}, raw-only (tax) {only_raw}, memory-only (benefit) {only_mem}, n={len(by)}\")\n    mem_tag = a.tag if a.memory_from is None else a.memory_from\n    mems = jl_read(os.path.join(out_dir(mem_tag), \"memory_units.jsonl\"))\n    nf = [sum(len(u[\"facts\"]) for u in m[\"units\"]) for m in mems]\n    print(f\"Memory units per question: mean {sum(nf) / max(len(nf), 1):.1f}\")\n    lens = collections.defaultdict(list)\n    for r in rows:\n        lens[r[\"context\"]].append(r[\"ctx_chars\"])\n    print(\"Mean context chars: \" + \", \".join(f\"{c} {sum(v) / len(v):.0f}\" for c, v in lens.items()))\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"cmd\", choices=[\"build\", \"answer\", \"judge\", \"report\"])\n    ap.add_argument(\"--n\", type=int, default=60)\n    ap.add_argument(\"--seed\", type=int, default=0)\n    ap.add_argument(\"--builder\", default=\"Qwen/Qwen3-1.7B\")\n    ap.add_argument(\"--answerer\", default=\"Qwen/Qwen3-1.7B\")\n    ap.add_argument(\"--judge\", default=\"Qwen/Qwen3-4B\")\n    ap.add_argument(\"--max-new\", type=int, default=400)\n    ap.add_argument(\"--tag\", default=\"\", help=\"results subfolder; empty = results/\")\n    ap.add_argument(\"--memory-from\", default=None,\n                    help=\"tag whose memory_units.jsonl to read (writer/reader swap); default = --tag\")\n    a = ap.parse_args()\n    {\"build\": cmd_build, \"answer\": cmd_answer, \"judge\": cmd_judge, \"report\": cmd_report}[a.cmd](a)\n\nif __name__ == \"__main__\":\n    main()\n```\n\n*Share your thoughts in the comments — I'd love to hear how this technology is impacting your industry.*", "url": "https://wpnews.pro/news/three-papers-say-your-agent-s-memory-loses-to-plain-rag", "canonical_source": "https://zyvop.com/three-papers-say-your-agent-s-memory-loses-to-plain-rag-vq2dm", "published_at": "2026-11-04 12:30:00+00:00", "updated_at": "2026-11-04 12:50:55.234499+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "artificial-intelligence"], "entities": ["He et al.", "UIUC", "Tsinghua", "LongMemEval-s", "Qwen3.6-27B", "A-Mem", "OpenViking", "MAGMA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/three-papers-say-your-agent-s-memory-loses-to-plain-rag", "markdown": "https://wpnews.pro/news/three-papers-say-your-agent-s-memory-loses-to-plain-rag.md", "text": "https://wpnews.pro/news/three-papers-say-your-agent-s-memory-loses-to-plain-rag.txt", "jsonld": "https://wpnews.pro/news/three-papers-say-your-agent-s-memory-loses-to-plain-rag.jsonld"}}