Everyone is building AI memory systems.
But how do we know which ones actually work?
As AI agents move from one-off interactions toward long-term collaboration, memory is becoming a core capability. Yet evaluating memory systems fairly is surprisingly difficult.
Different systems often use different datasets, answer models, prompts, and evaluation methods. When the final score changes, it can be hard to tell whether the difference comes from the memory system itself or from the evaluation setup.
That's why we built Agent Memory Leaderboard (AML).
Why Do We Need a Memory Benchmark?
Memory is more than storing conversation history.
A useful memory system needs to retrieve relevant information, connect information across time, handle changing states, and provide useful context for an agent's current task.
But there hasn't been a common evaluation environment where different memory approaches can be compared under the same conditions.
AML was created to provide that common ground.
The first edition was jointly initiated by nearly 30 universities and research institutions and covers two evaluation tracks:
As of August 12, 2026:
136 teams registered for the competition
67 representative memory frameworks successfully completed the first evaluation
The AML website surpassed 200,000 clicks
The website passed 100,000 clicks within the first 10 days
The first leaderboard results are now live.
Making Memory Systems More Comparable
One of the main challenges we wanted to address was evaluation consistency.
In a typical setup, a memory system may be evaluated together with a particular answer model, prompt, or judge.
That makes direct comparison difficult.
A higher score could come from a better memory system — but it could also come from a stronger downstream model or a different evaluation setup.
AML tries to separate these components.
The core interface for participating memory systems is:
Memory System
Add → Search
The memory system receives long-term history through Add, and returns relevant memories through Search.
Then AML handles:
AML Evaluation Platform
Answer → Eval
Answer generation and evaluation are completed by the benchmark platform under the same evaluation process.
This helps reduce the impact of different answer models, prompts, judges, and scoring conventions.
The goal is simple:
Compare memory systems under the same conditions as much as possible.
Memory Is More Than Retrieval
A memory system shouldn't be judged only by whether it can retrieve something that looks similar. Memory quality is not only about retrieving similar information, but about understanding relevance, context, time, and task requirements.
For text memory, AML evaluates multiple capabilities, including: This matters because real-world agent memory is not just a search problem.
An agent may need to understand:
A useful memory system needs to handle these questions together.
The First Results
The first AML evaluation has now been completed.
67 representative memory frameworks successfully completed the evaluation across two tracks covering text memory for both open-source methods and commercial products.
The complete rankings, scores, and system versions are available on the leaderboard.
Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual ] What Comes Next?
The first leaderboard is not the finish line.
We plan to keep AML running as a long-term evaluation and public leaderboard for agent memory systems.
Going forward, we will publish deeper technical analyses of the first-round results, including:
We also want the benchmark itself to evolve.
If you are working on AI agents, memory systems, evaluation, or related research, we would love to hear what you think a useful memory benchmark should measure next. Four Evaluation Tracks
To better reflect different types of memory systems, AML organizes evaluation into two tracks:
Open-source Methods
Commercial Products
Each track evaluates systems under the same benchmark framework.
Explore AML
Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual] GitHub:
[https://github.com/AML-memory/agent-memory-leaderboard] Hugging Face Space:
[https://huggingface.co/agent-memory-leaderboard] X:
[https://x.com/AgentMemoryL/status/2087544165433590240] Contact:
contactus@agentmemoryleaderboard.ai The first results are out.
Now the real work begins: