{"slug": "how-nemori-helps-ai-agents-remember-what-matters", "title": "How NEMORI Helps AI Agents Remember What Matters.", "summary": "NEMORI, a framework detailed in the arXiv paper \"What Deserves Memory: Adaptive Memory Distillation for LLM Agents,\" builds long-term memory around a language model by distilling conversations into two memory types: episodic memory, which preserves an experience with its context, and semantic memory, which preserves standalone facts such as an employer or preference. NEMORI buffers complete messages with sender, content and timestamp, and in the paper's main configuration begins processing when the buffer holds 20 messages, using each message's position index to group related messages into episodes before the model organizes them into retrievable memories. The framework leaves the underlying model unchanged, developing stored information around it instead.", "body_md": "You’re catching up with a friend you haven’t seen in a few weeks. The conversation wanders through the usual things. Work has been busy. Someone has moved apartments. There’s a film you both meant to watch. Somewhere in the middle, your friend mentions starting a new job.\n\nYou ask how it’s going. She tells you about the team, what she’s working on, and why she finally decided to leave her previous company.\n\nThen the conversation moves on.\n\nA week later, when you speak again, you ask about the new job. It’s a small gesture. But a detail from an earlier conversation has found its way into the next one. You can pick up a thread without asking her to explain everything again.\n\nThat continuity makes a conversation feel familiar.\n\nOver time, those threads accumulate. You learn what someone cares about, what they are trying to do, and which parts of their life have changed. Your understanding develops across ordinary exchanges.\n\nA passing remark becomes something you know about a person. We increasingly expect something similar from AI assistants.\n\nWe explain our work, share preferences, discuss decisions, and return days later hoping the next conversation will build on the previous one. An assistant helping with a project should remember an earlier decision. A change in circumstances should influence its suggestions.\n\nMaking that happen takes more than preserving a chat history.\n\nA conversation contains repetition, temporary concerns, useful facts, and updates to things said earlier. Some details need their surrounding context. Others deserve to become information the assistant can retrieve directly.\n\n**NEMORI is a framework for building long-term memory around a language model.** It takes the conversations an assistant experiences, turns them into organized memories, and makes those memories available when a later conversation needs them.\n\nThe underlying model stays the same. What develops over time is the information stored around it.\n\nThe approach, introduced in [*What Deserves Memory: Adaptive Memory Distillation for LLM Agents*](https://arxiv.org/html/2508.03341v4), maintains two related forms of memory.\n\nAn **episodic memory** preserves an experience: what happened, who was involved, and the context surrounding it. It is something like a diary entry.\n\nA **semantic memory** preserves knowledge that can stand on its own: someone’s employer, a preference, a project decision, or the reason behind that decision.\n\nKeeping both gives the assistant a clear account of an experience alongside facts it can find directly.\n\nSuppose an assistant already knows that Alice works at Acme and prefers Python. On May 10, she shares an update.\n\nIncoming messages enter a temporary buffer. Each element is a complete message containing its sender, content, and timestamp. User and assistant messages count as separate elements.\n\nIn the paper’s main configuration, processing begins when the buffer contains **20 messages**:\n\n```\nbuffer = [    m1,  m2,  m3,  m4,  m5,    m6,  m7,  m8,  m9,  m10,    m11, m12, m13, m14, m15,    m16, m17, m18, m19, m20,]\n```\n\nEach message receives a number corresponding to its position in this window: m1 is message 1, m2 is message 2, and so on.\n\nSuppose the assistant already knows that Alice works at Acme and prefers Python. The beginning of the buffered conversation might look like this:\n\n```\nDate: May 10, 20261. Alice: I now work at Nova. I started yesterday.2. Alice: Also, I’m thinking about a holiday next month.3. Assistant: What role?4. Alice: Backend engineer. Our team uses Go because deployment is simpler.\n```\n\nThe remaining messages continue the holiday discussion and later move into debugging a SQL query.\n\nThe buffer therefore contains several discussions, with some messages interleaved. Their position numbers give the next stage a way to identify which messages belong together.\n\n2. **Group related messages into episodes using their indices**\n\nNEMORI gives the language model the numbered messages and asks it to organize them into coherent episodes.\n\nA **group** is a collection of messages belonging to the same event or discussion. The model returns the message numbers — called **indices** — that identify its members.\n\nFor example:\n\n```\ngroups = [    {        \"topic\": \"Alice's new job\",        \"indices\": [1, 3, 4],    },    {        \"topic\": \"Holiday plans\",        \"indices\": [2, 5, 6, 7],    },    {        \"topic\": \"Debugging a SQL query\",        \"indices\": [            8, 9, 10, 11, 12, 13, 14,            15, 16, 17, 18, 19, 20,        ],    },]\n```\n\nHere, [1, 3, 4] means that the job episode contains the original messages m1, m3, and m4.\n\nThe indices can be nonconsecutive because topics can be interleaved. Message 2 belongs to the holiday discussion, even though the conversation returns to Alice\n\n3. **Write a narrative and a cue for each group**\n\nThe program uses each group’s indices to retrieve its actual messages:\n\n```\ngroup = groups[0]raw_messages = [    buffer[index - 1]    for index in group[\"indices\"]]# For indices [1, 3, 4]:# raw_messages contains [m1, m3, m4]\n```\n\nThe subtraction is simply because the message labels start at 1, while Python list positions start at 0.\n\nThese are the complete original message records, including their timestamps. For the job group, they contain Alice’s update about Nova, the question about her role, and her answer about backend engineering and Go.\n\nFor each episode, A language model then creates a **narrative** and a short description called a **cue** from those selected messages.\n\nFor Alice’s job discussion, the resulting record might look like this:\n\n```\nepisode = {    \"cue\": \"Alice discusses a new role and the tools used by her team\",    \"narrative\": (        \"On May 10, 2026, Alice explained that she joined Nova \"        \"on May 9 as a backend engineer. Her team uses Go \"        \"because it simplifies deployment.\"    ),    \"raw_messages\": conversation,}\n```\n\nThe narrative connects Alice’s statements and resolves “yesterday” into May 9. The original messages remain attached, preserving the conversation behind the rewritten account.\n\nThis process repeats for each group. The holiday messages produce their own episode, and the SQL discussion produces another.\n\n4. **Make the episode searchable**\n\nNEMORI creates an **embedding** from the cue and narrative — a numerical representation that helps it find related meanings.\n\n```\nepisode[\"embedding\"] = embed(    episode[\"cue\"] + \"\\n\" + episode[\"narrative\"])\n```\n\nA later question about Alice’s employer can therefore find this episode even if it uses different wording.\n\n5. **Check whether it continues an existing event**\n\nA discussion can stretch across separate processing windows.\n\nNEMORI retrieves related episodic memories and asks a language model whether one belongs to the same event. If it finds a continuation, it combines the narratives, retains the source messages, and updates the embedding.\n\nThis helps reconnect an experience that processing boundaries initially split apart.\n\n6. **Store the completed episode**\n\nThe episode is saved in episodic memory.\n\n```\nepisodic_memory.save(episode)\n```\n\nThe assistant now has a coherent account of Alice’s update, its original messages, and a way to find it.\n\n*The next four steps examine what useful knowledge this experience adds.*\n\n7. **Retrieve relevant existing knowledge**\n\nNEMORI searches its semantic memory using the new episode’s embedding.\n\n```\nknown_facts = search_semantic_memory(    episode[\"embedding\"])\n```\n\nSuppose the search returns:\n\n```\nknown_facts = [    \"Alice currently works at Acme.\",    \"Alice prefers Python.\",]\n```\n\nThese facts represent relevant pieces of the assistant’s earlier understanding. They become context for a prediction about the new episode.\n\n8. **Predict the episode from limited clues**\n\nA language model receives the cue from Alice’s new episode together with the retrieved facts.\n\n```\nprediction = predict_episode(    cue=episode[\"cue\"],    known_facts=known_facts,)\n```\n\nThe full narrative and original messages are withheld from this call. From the limited clues, the model might produce:\n\n*Alice discussed changes at work and her experience using Python.*\n\nThe prediction misses Nova, the joining date, and the team’s use of Go. Those omissions give NEMORI something useful to examine.\n\n9. **Extract useful gaps**\n\nAnother language-model call compares the prediction with the original conversation.\n\n```\nnew_facts = extract_useful_differences(    actual=episode[\"raw_messages\"],    predicted=prediction,)\n```\n\nIts task is to identify useful information that the prediction omitted or misrepresented.\n\nA possible result is:\n\n```\nnew_facts = [    \"Alice currently works at Nova as a backend engineer.\",    \"Alice joined Nova on May 9, 2026.\",    \"Alice's team uses Go because it simplifies deployment.\",]\n```\n\nThis is **prediction error distillation**.\n\nExisting knowledge helps establish an expectation. The actual conversation reveals where that expectation falls short. Useful gaps become candidates for new semantic memories.\n\nThe facts still come from Alice’s conversation. The prediction provides a baseline for deciding what the experience has added.\n\nThe extraction prompt also asks the model to favor specific information useful for future interactions. Pleasantries, conversational style, and temporary reactions generally do not need separate entries.\n\n10. **Reconcile the new facts with memory**\n\nNEMORI compares each candidate with related stored facts. It decides whether to add a new entry, combine overlapping knowledge, or replace information invalidated by a contradiction.\n\n```\nfor fact in new_facts:    related_facts = semantic_memory.search(embed(fact))    decision = decide_memory_update(        new_fact=fact,        existing_facts=related_facts,    )    semantic_memory.apply(decision)\n```\n\nAlice’s undated current-employer entry can change from Acme to Nova. Her preference for Python can remain.\n\nA team’s choice of Go and an individual’s preference for Python describe different things. Both can be true, and the update needs to preserve that distinction.\n\nAfter these steps, the assistant retains two useful views of the conversation.\n\nThe **episodic memory** preserves Alice’s job discussion, including its narrative and original messages. The **semantic memory** contains standalone facts about her employer, role, joining date, and team’s tools.\n\nWhen a question arrives later, NEMORI searches both stores. Relevant narratives, facts, and selected original messages become context for the answer-generating model.\n\n```\ncontext = {    \"episodes\": search_episodic_memory(question),    \"facts\": search_semantic_memory(question),}answer = generate_answer(question, context)\n```\n\nIf Alice asks when she joined Nova, the system can retrieve the date. If she asks for help with a service at work, it can retrieve the team’s use of Go and the reason behind that choice.\n\nThe assistant can draw on a concise fact while retaining access to the experience that supports it.\n\n**A useful part of this architecture is that some of the work happens before the next question arrives.**\n\nThe system has already grouped related messages, clarified the event, and extracted information that may matter later. A future answer can benefit from that preparation.\n\nIt also preserves different levels of detail. A straightforward question may need one semantic fact. A more involved question may need the surrounding episode or the original conversation.\n\nAnd its understanding can evolve as new information arrives. Alice’s employer changes, while other knowledge about her remains valid. As the assistant’s knowledge develops, its expectations can change, and different parts of a new experience may stand out.\n\nThe process still depends on the model’s judgments. A prediction can miss information the system already knows or guess a new detail correctly. Predictability is a useful signal for finding knowledge gaps, rather than a perfect measure of future importance.\n\nRetaining the source conversation gives the system evidence it can revisit.\n\nThe practical promise is that an assistant can carry forward useful knowledge while supplying a smaller, more focused context for each new question.\n\nThe paper’s benchmarks offer encouraging evidence for that promise.\n\nOn **LoCoMo**, with GPT-4o-mini, NEMORI achieved an average LLM-judge score of **73.0**, compared with **72.3** for full conversation history. Its average answer context was approximately **88% smaller** — 2,745 tokens instead of 23,653.\n\nOn the longer **LongMemEval-S** benchmark, with GPT-4.1-mini, the average score increased from **65.6 with full history to 74.6**. Across the reported models on that benchmark, the context supplied for answering was approximately **95–96% smaller**.\n\nUnder those experimental settings, carefully constructed memories supported similar or better average judged answers while giving the model substantially less history to read\n\nI like playing around with memory systems — what agents should remember, how those memories shape their context, and how experience can make them more useful over time. There’s plenty here I’m still figuring out, and I enjoy sharing what I learn along the way.\n\n**conclusion**\n\nI like playing around with memory systems ,what agents should remember, how those memories shape their context, and how experience can make them more useful over time. There’s plenty here I’m still figuring out, and I enjoy sharing what I learn along the way.\n\nIf you’re interested in **applied AI, context engineering, and self-improving agents**, follow along. I’ll be writing more about the ideas I’m exploring and the experiments I’m trying.\n\n[How NEMORI Helps AI Agents Remember What Matters.](https://pub.towardsai.net/how-nemori-helps-ai-agents-remember-what-matters-0d0a2d6588b6) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-nemori-helps-ai-agents-remember-what-matters", "canonical_source": "https://pub.towardsai.net/how-nemori-helps-ai-agents-remember-what-matters-0d0a2d6588b6?source=rss----98111c9905da---4", "published_at": "2026-10-22 23:01:02+00:00", "updated_at": "2026-10-22 23:12:26.935838+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "artificial-intelligence", "ai-research"], "entities": ["NEMORI", "What Deserves Memory: Adaptive Memory Distillation for LLM Agents", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-nemori-helps-ai-agents-remember-what-matters", "markdown": "https://wpnews.pro/news/how-nemori-helps-ai-agents-remember-what-matters.md", "text": "https://wpnews.pro/news/how-nemori-helps-ai-agents-remember-what-matters.txt", "jsonld": "https://wpnews.pro/news/how-nemori-helps-ai-agents-remember-what-matters.jsonld"}}