# MemDelta Shakes Up AI Memory Evaluation with Surprising Results > Source: > Published: 2026-06-30 19:23:54+00:00 # MemDelta Shakes Up AI Memory Evaluation with Surprising Results MemDelta offers a fresh evaluation protocol for AI memory systems, revealing unexpected performance dynamics across models. Why should we care? Because it challenges current assumptions. Evaluating AI memory systems has always been a tricky endeavor, often clouded by mixed results due to concurrent changes in multiple components like language models or retrieval pipelines. Enter MemDelta, a new protocol that promises clarity by altering just one component at a time. It's like a controlled experiment in a field prone to chaos. ## The [RAG](/glossary/rag) vs. Full-Context Battle AI, Retrieval-Augmented Generation (RAG) models often go head-to-head with full-context systems like [GPT](/glossary/gpt)-4o-mini. MemDelta's findings take an unexpected turn here. On one hand, verbatim RAG matches full-context performance with a narrow margin (47.2% to 49.8%). Flip the lens, though, and you see [Gemini](/glossary/gemini) gaining a notable 14 percentage points from full-context, while Sonnet does the opposite, adding 31 points with RAG. It's a classic case of results heavily dependent on the model family. [Embedding](/glossary/embedding) Model's Subtle Yet Significant Influence Consider a seemingly minor swap: changing just the embedding model. The result? A six percentage point swing in accuracy at a sample size of 500. Mem0 outpaces MiniLM-RAG by a hefty 11 points but falls just short of cloud-RAG by 1.2 points. It begs the question: how often are conclusions drawn without this level of detail? ## Agent Memory vs. Basic Retrieval Agent self-memory, often touted as the future, underperforms when compared to basic retrieval methods, scoring 42% against 47%. On the surface, it's a small gap, but it underlines an important point: is agent memory being oversold? on two out of six question types, Mem0 matches the performance of cloud RAG at a staggering 50 times the cost. This narrowing of gains rather than a broad improvement suggests a need to recalibrate our expectations. ## Recommendations for Future Evaluations MemDelta's insights come with recommendations that aim to refine future evaluations. Fixing embedding models across comparisons, stratifying by model family, and reporting write-path costs are all on the table. These steps could help ensure that any attributed architectural gains are legitimate rather than coincidental. The AI-AI Venn diagram is getting thicker, and ignoring these nuances might lead to misguided investments in memory systems that don't deliver sustainable benefits. The [compute](/glossary/compute) layer needs a payment rail, and MemDelta could be the first step towards that. Get AI news in your inbox Daily digest of what matters in AI.