You’re catching up with a friend you haven’t seen in a few weeks. The conversation wanders through the usual things. Work has been busy. Someone has moved apartments. There’s a film you both meant to watch. Somewhere in the middle, your friend mentions starting a new job.
You ask how it’s going. She tells you about the team, what she’s working on, and why she finally decided to leave her previous company.
Then the conversation moves on.
A week later, when you speak again, you ask about the new job. It’s a small gesture. But a detail from an earlier conversation has found its way into the next one. You can pick up a thread without asking her to explain everything again.
That continuity makes a conversation feel familiar.
Over time, those threads accumulate. You learn what someone cares about, what they are trying to do, and which parts of their life have changed. Your understanding develops across ordinary exchanges.
A passing remark becomes something you know about a person. We increasingly expect something similar from AI assistants.
We explain our work, share preferences, discuss decisions, and return days later hoping the next conversation will build on the previous one. An assistant helping with a project should remember an earlier decision. A change in circumstances should influence its suggestions.
Making that happen takes more than preserving a chat history.
A conversation contains repetition, temporary concerns, useful facts, and updates to things said earlier. Some details need their surrounding context. Others deserve to become information the assistant can retrieve directly.
NEMORI is a framework for building long-term memory around a language model. It takes the conversations an assistant experiences, turns them into organized memories, and makes those memories available when a later conversation needs them.
The underlying model stays the same. What develops over time is the information stored around it.
The approach, introduced in What Deserves Memory: Adaptive Memory Distillation for LLM Agents, maintains two related forms of memory.
An episodic memory preserves an experience: what happened, who was involved, and the context surrounding it. It is something like a diary entry.
A semantic memory preserves knowledge that can stand on its own: someone’s employer, a preference, a project decision, or the reason behind that decision.
Keeping both gives the assistant a clear account of an experience alongside facts it can find directly.
Suppose an assistant already knows that Alice works at Acme and prefers Python. On May 10, she shares an update.
Incoming messages enter a temporary buffer. Each element is a complete message containing its sender, content, and timestamp. User and assistant messages count as separate elements.
In the paper’s main configuration, processing begins when the buffer contains 20 messages:
buffer = [ m1, m2, m3, m4, m5, m6, m7, m8, m9, m10, m11, m12, m13, m14, m15, m16, m17, m18, m19, m20,]
Each message receives a number corresponding to its position in this window: m1 is message 1, m2 is message 2, and so on.
Suppose the assistant already knows that Alice works at Acme and prefers Python. The beginning of the buffered conversation might look like this:
Date: May 10, 20261. Alice: I now work at Nova. I started yesterday.2. Alice: Also, I’m thinking about a holiday next month.3. Assistant: What role?4. Alice: Backend engineer. Our team uses Go because deployment is simpler.
The remaining messages continue the holiday discussion and later move into debugging a SQL query.
The buffer therefore contains several discussions, with some messages interleaved. Their position numbers give the next stage a way to identify which messages belong together.
- Group related messages into episodes using their indices
NEMORI gives the language model the numbered messages and asks it to organize them into coherent episodes.
A group is a collection of messages belonging to the same event or discussion. The model returns the message numbers — called indices — that identify its members.
For example:
groups = [ { "topic": "Alice's new job", "indices": [1, 3, 4], }, { "topic": "Holiday plans", "indices": [2, 5, 6, 7], }, { "topic": "Debugging a SQL query", "indices": [ 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, ], },]
Here, [1, 3, 4] means that the job episode contains the original messages m1, m3, and m4.
The indices can be nonconsecutive because topics can be interleaved. Message 2 belongs to the holiday discussion, even though the conversation returns to Alice
- Write a narrative and a cue for each group
The program uses each group’s indices to retrieve its actual messages:
group = groups[0]raw_messages = [ buffer[index - 1] for index in group["indices"]]# For indices [1, 3, 4]:# raw_messages contains [m1, m3, m4]
The subtraction is simply because the message labels start at 1, while Python list positions start at 0.
These are the complete original message records, including their timestamps. For the job group, they contain Alice’s update about Nova, the question about her role, and her answer about backend engineering and Go.
For each episode, A language model then creates a narrative and a short description called a cue from those selected messages.
For Alice’s job discussion, the resulting record might look like this:
episode = { "cue": "Alice discusses a new role and the tools used by her team", "narrative": ( "On May 10, 2026, Alice explained that she joined Nova " "on May 9 as a backend engineer. Her team uses Go " "because it simplifies deployment." ), "raw_messages": conversation,}
The narrative connects Alice’s statements and resolves “yesterday” into May 9. The original messages remain attached, preserving the conversation behind the rewritten account.
This process repeats for each group. The holiday messages produce their own episode, and the SQL discussion produces another.
- Make the episode searchable
NEMORI creates an embedding from the cue and narrative — a numerical representation that helps it find related meanings.
episode["embedding"] = embed( episode["cue"] + "\n" + episode["narrative"])
A later question about Alice’s employer can therefore find this episode even if it uses different wording.
- Check whether it continues an existing event
A discussion can stretch across separate processing windows.
NEMORI retrieves related episodic memories and asks a language model whether one belongs to the same event. If it finds a continuation, it combines the narratives, retains the source messages, and updates the embedding.
This helps reconnect an experience that processing boundaries initially split apart.
- Store the completed episode
The episode is saved in episodic memory.
episodic_memory.save(episode)
The assistant now has a coherent account of Alice’s update, its original messages, and a way to find it.
The next four steps examine what useful knowledge this experience adds.
- Retrieve relevant existing knowledge
NEMORI searches its semantic memory using the new episode’s embedding.
known_facts = search_semantic_memory( episode["embedding"])
Suppose the search returns:
known_facts = [ "Alice currently works at Acme.", "Alice prefers Python.",]
These facts represent relevant pieces of the assistant’s earlier understanding. They become context for a prediction about the new episode.
- Predict the episode from limited clues
A language model receives the cue from Alice’s new episode together with the retrieved facts.
prediction = predict_episode( cue=episode["cue"], known_facts=known_facts,)
The full narrative and original messages are withheld from this call. From the limited clues, the model might produce:
Alice discussed changes at work and her experience using Python.
The prediction misses Nova, the joining date, and the team’s use of Go. Those omissions give NEMORI something useful to examine.
- Extract useful gaps
Another language-model call compares the prediction with the original conversation.
new_facts = extract_useful_differences( actual=episode["raw_messages"], predicted=prediction,)
Its task is to identify useful information that the prediction omitted or misrepresented.
A possible result is:
new_facts = [ "Alice currently works at Nova as a backend engineer.", "Alice joined Nova on May 9, 2026.", "Alice's team uses Go because it simplifies deployment.",]
This is prediction error distillation.
Existing knowledge helps establish an expectation. The actual conversation reveals where that expectation falls short. Useful gaps become candidates for new semantic memories.
The facts still come from Alice’s conversation. The prediction provides a baseline for deciding what the experience has added.
The extraction prompt also asks the model to favor specific information useful for future interactions. Pleasantries, conversational style, and temporary reactions generally do not need separate entries.
- Reconcile the new facts with memory
NEMORI compares each candidate with related stored facts. It decides whether to add a new entry, combine overlapping knowledge, or replace information invalidated by a contradiction.
for fact in new_facts: related_facts = semantic_memory.search(embed(fact)) decision = decide_memory_update( new_fact=fact, existing_facts=related_facts, ) semantic_memory.apply(decision)
Alice’s undated current-employer entry can change from Acme to Nova. Her preference for Python can remain.
A team’s choice of Go and an individual’s preference for Python describe different things. Both can be true, and the update needs to preserve that distinction.
After these steps, the assistant retains two useful views of the conversation.
The episodic memory preserves Alice’s job discussion, including its narrative and original messages. The semantic memory contains standalone facts about her employer, role, joining date, and team’s tools.
When a question arrives later, NEMORI searches both stores. Relevant narratives, facts, and selected original messages become context for the answer-generating model.
context = { "episodes": search_episodic_memory(question), "facts": search_semantic_memory(question),}answer = generate_answer(question, context)
If Alice asks when she joined Nova, the system can retrieve the date. If she asks for help with a service at work, it can retrieve the team’s use of Go and the reason behind that choice.
The assistant can draw on a concise fact while retaining access to the experience that supports it.
A useful part of this architecture is that some of the work happens before the next question arrives.
The system has already grouped related messages, clarified the event, and extracted information that may matter later. A future answer can benefit from that preparation.
It also preserves different levels of detail. A straightforward question may need one semantic fact. A more involved question may need the surrounding episode or the original conversation.
And its understanding can evolve as new information arrives. Alice’s employer changes, while other knowledge about her remains valid. As the assistant’s knowledge develops, its expectations can change, and different parts of a new experience may stand out.
The process still depends on the model’s judgments. A prediction can miss information the system already knows or guess a new detail correctly. Predictability is a useful signal for finding knowledge gaps, rather than a perfect measure of future importance.
Retaining the source conversation gives the system evidence it can revisit.
The practical promise is that an assistant can carry forward useful knowledge while supplying a smaller, more focused context for each new question.
The paper’s benchmarks offer encouraging evidence for that promise.
On LoCoMo, with GPT-4o-mini, NEMORI achieved an average LLM-judge score of 73.0, compared with 72.3 for full conversation history. Its average answer context was approximately 88% smaller — 2,745 tokens instead of 23,653.
On the longer LongMemEval-S benchmark, with GPT-4.1-mini, the average score increased from 65.6 with full history to 74.6. Across the reported models on that benchmark, the context supplied for answering was approximately 95–96% smaller.
Under those experimental settings, carefully constructed memories supported similar or better average judged answers while giving the model substantially less history to read
I like playing around with memory systems — what agents should remember, how those memories shape their context, and how experience can make them more useful over time. There’s plenty here I’m still figuring out, and I enjoy sharing what I learn along the way.
conclusion
I like playing around with memory systems ,what agents should remember, how those memories shape their context, and how experience can make them more useful over time. There’s plenty here I’m still figuring out, and I enjoy sharing what I learn along the way.
If you’re interested in applied AI, context engineering, and self-improving agents, follow along. I’ll be writing more about the ideas I’m exploring and the experiments I’m trying.
How NEMORI Helps AI Agents Remember What Matters. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.