Benchmarking continual learning through memory.
Models #
Which models are the best at memory generation?
| Rank | Model | Score | Task cost | |
|---|---|---|---|---|
| 1 | Opus 5.5 [max] | 73.3% | $2.80 | |
| 2 | GLM-5.3 [max] | 73.3% | $3.58 | |
| 3 | Kimi K3 [max] | 66.7% | $3.73 | |
| 4 | GPT-6.1 Sol [max] | 60.0% | $0.77 | |
| 5 | Gemini 4 Argon [high] | 60.0% | $2.60 | |
| 6 | Muse Wonder [xhigh] | 46.7% | $4.09 | |
| 7 | Grok 4.7 [xhigh] | 33.3% | $3.02 |
-
GPT-6.1 Sol
-
Opus 5.5
-
Muse Wonder
-
Gemini 4 Argon
-
GLM-5.3
-
Kimi K3
-
Grok 4.7
-
Task effort:
-
low
-
medium
-
high
-
xhigh
-
max
Harnesses #
Which harnesses are the best at memory generation?
| Rank | Harness | Score | Task cost | |
|---|---|---|---|---|
| 1 | Letta Dream | 73.3% | $2.80 | |
| 2 | Prime Agent | 60.0% | $3.74 | |
| 3 | OpenAI Agents SDK | 40.0% | $0.82 | |
| 4 | Claude Managed Agents | 33.3% | $3.21 | |
| 5 | Hermes Agent | 26.7% | $2.79 |
-
Letta Dream
-
OpenAI Agents SDK
-
Hermes Agent
-
Prime Agent
-
Claude Managed Agents
-
Task effort:
-
low
-
medium
-
high
-
xhigh
-
max
The first memory-native benchmark #
Existing memory benchmarks measure recall of past facts. FrontierMemory is the first benchmark that measures the quality of memory. Agents are given access to trajectories of past experience and tasked to generate memory, which is then scored by how much it helps an agent solve downstream tasks.
About FrontierMemory #
Agents are given access to trajectories of agent experience and generate memory. 2. Memory is evaluated based on how much it helps solve downstream tasks.
Task sources #
We select challenging coding and knowledge-work tasks from public benchmarks, including HarborIndex, DeepSWE, and SkillsBench, both for source transcripts and for downstream tasks. Some tasks are modified so solving them requires knowledge available only in the source transcripts.
Sample tasks #
DeepSWEAwilix Release Tier (Implicit)
Add async initialization to Awilix, where the version bump is determined by applying a policy learned from earlier tasks to a release tier found in project metadata.
BigCodeBenchWord2Vec Pipeline
Build a Word2Vec component whose interface, preprocessing, and output contract must be inferred from previous runs.
SkillsBenchNetwork Intrusion Analysis
Compute DAPT2020 network statistics using lessons from failed analyses about counting, entropy, and threat detection.
Questions and answers #
What is the final metric used for the leaderboard? #
Each memory-generation harness and model is ranked by the higher downstream task solve rate of Opus 5.5 and GPT-6 Sol, both at max reasoning. When only one task model is available, its score is used.
What is the time limit for memory generation? #
Memory generation is limited to 6 hours.
How does FrontierMemory differ from other memory benchmarks? #
Existing memory benchmarks mostly measure recall of past facts. FrontierMemory is the first benchmark to measure memory by how much it helps an agent solve real tasks.
What does the agent see during memory generation? #
The agent sees the source transcripts. These cover the downstream tasks as well as other, unrelated tasks, and each transcript is labeled with whether the attempt succeeded or failed.
Are agents allowed to use the internet? #
No. Web browsing and online solution lookup are not allowed during memory generation or task execution. Agents work from the provided trajectories and task environment; model API calls remain available.
What goes in a submission? #
FrontierMemory primarily compares models on how well they generate memory. By default, each model generates memory in the Letta Dream harness and is evaluated in Letta Code. Submissions can also bring their own memory generation and task harnesses, to maximize compatibility and allow harness comparisons.
Submit an agent harness or model #
To submit a harness, it must have a memory system it uses to create memories or skills between sessions.
Sign up for leaderboard updates #
Get notified when new harnesses and models are added to FrontierMemory. To reach our team, email [email protected].