MemDelta

MemDelta, a controlled evaluation protocol for agent memory systems, reveals that reported performance gains often stem from changes in embedding models or language models rather than memory architecture itself. Testing on LongMemEval-S across three model families, the study found that swapping only the embedding model shifts accuracy by up to 6.2 percentage points, and that agent self-memory underperforms basic retrieval. The authors recommend fixing embedding models and stratifying by model family in future evaluations.

Agent memory systems are increasingly evaluated against RAG and full-context baselines, but reported gains often mix changes in the memory method with changes in the language model, embedding model, or retrieval pipeline, making it unclear what is actually being measured. We present MemDelta, a controlled evaluation protocol that varies one component at a time on LongMemEval-S 500 questions, 50+ sessions, three model families . Four findings emerge: 1 verbatim RAG matches full-context GPT-4o-mini 47.2% vs. 49.8%, p = 0.34 , but the ranking reverses across models: Gemini gains +14pp from full context, while Sonnet gains +31pp from RAG, partly because it refuses 63% of full-context queries; 2 swapping only the embedding model in an identical pipeline shifts accuracy by +6.2pp at n = 500 p = 0.004 , and Mem0 beats MiniLM-RAG by +11pp but loses to cloud-RAG by 1.2pp, so one variable flips the conclusion; 3 agent self-memory 42% underperforms basic retrieval 47% ; 4 on 2 of 6 question types n = 88 , Mem0 matches cloud RAG 72.7% vs. 73.9%, p = 1.0 at 50x the cost, suggesting narrow rather than general gains. We recommend memory evaluations fix embedding models across comparisons, stratify by model family, and report write-path cost before attributing gains to architecture. Category: Uncategorized. Imported rows: 6. Top imported result: S4b: Verbatim RAG cloud , rank 1, 53.40.