This series of posts is designed to bring you the newest findings and developments in the NLP field. I’ll delve into four significant research papers each month, offering a comprehensive summary. Be sure to visit my blog regularly or subscribe to my newsletter for monthly updates. Let’s dive in!
📝 Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models [paper] [* code**]*
Engram gives a language model a simple memory system for common local patterns, so it does not have to rebuild the same information through many neural layers. It does this by turning short groups of nearby tokens into keys that retrieve learned vectors from a large memory table.
Before Engram, language models stored knowledge mainly inside their neural network weights. To recognize common phrases or entities, the model had to reconstruct that information through several Transformer layers. Mixture-of-Experts made this more efficient by activating only a small set of experts for each token, but it was still using computation to recover memorized information. In other words, the model was often “thinking” through facts or patterns that could have been retrieved directly from memory.
Engram adds a large lookup table alongside the Transformer. It uses 2-grams and 3-grams, hashes them, and combines several lookup results. A gate decides how much of the retrieved information should be used during computations. The module is placed at selected layers (2 and 15) rather than everywhere.
A key novelty is showing that compute and memory should be split instead of giving everything to MoE experts. They found the best mix was roughly 80% MoE and 20% Engram in their tests. They showed that Engram can store much more information without making each token much more expensive to process. Even if the memory table becomes huge, the model still looks up only a small number of entries for each token. Finally, because the lookup addresses are known early, the memory can be fetched from CPU memory with very little speed loss.
Engram-27B beat the same-size MoE model on many tasks, for example MMLU 57.4 → 60.4, BBH 50.9 → 55.9, and HumanEval 37.8 → 40.8. It also improved long-context retrieval, with Multi-Query NIAH rising from 84.2 to 97.0.
This is impressive because they removed some MoE expert parameters and replaced them with Engram memory, keeping both models at 26.7B total parameters while still getting better results.
📝 Zep: A Temporal Knowledge Graph Architecture for Agent Memory [paper]
This paper is a bit older, but I’m interested in memory systems right now, so I wanted to take a look at it.
As we know, AI agents often struggle to remember important information from long conversations and changing data. This makes it difficult for them to give accurate, consistent answers over time.
The main idea of this paper is to give an AI agent a memory that can keep old information, new information, and changes over time. The first level is the episode level, which stores the original messages, such as “I started a new job last week.” The second level is the entity and fact level, which stores important people, things, and relationships, such as “Alice works at Acme.” The third level is the community level, which groups related information together and makes a short summary, such as a group about Alice’s work history. Together, these three levels help the system keep detailed memories while also organizing them into useful facts and bigger topics.
They search for useful memories using vector similarity search, BM25 full-text search, and breadth-first search (BFS) over the graph. Then they rerank the results so the most relevant memories come first. Finally, they turn the best facts, entities, and dates into a short context for the AI.
In LongMemEval, Zep improved accuracy from 55.4% to 63.8% with GPT-4o-mini and from 60.2% to 71.2% with GPT-4o. It also cut latency from about 29–31 seconds to 2.6–3.2 seconds, while reducing the context from 115,000 tokens to about 1,600 tokens.
As impressive as it sounds, one caveat is that this is a private company paper about its own system, so the results are not from an independent evaluation.
📝 Memory Mosaics at scale [paper] This paper introduces Memory Mosaics v2, a new language model architecture that uses associative memory instead of traditional Transformer attention. It explores how this approach can help AI models store and retrieve information and learn new tasks from examples.
The architecture has three types of memory: short-term, long-term, and persistent memory. Short-term memory focuses on recent information, while long-term memory stores information from earlier parts of the text. Persistent memory keeps the general knowledge learned during training. The model uses attention-like scores to find and retrieve relevant information from its memories. It also introduces adaptive memory matching and learned gates to improve how information is stored and retrieved.
Short-term and long-term memory are two specialized attention-like mechanisms operating over different parts of the context, while persistent memory is a standard neural network that stores knowledge in its parameters.
Memory Mosaics v2 matches Transformers on standard language benchmarks, with both scoring 52.2% on average. It shines at remembering new information from long texts, scoring 53.4% compared to 41.1% for Transformers on 32K-token tasks. It also learns new tasks better from examples, with improvements of more than 10 percentage points on some classification tasks. Even with 8 times more training data, Transformers still fall behind Memory Mosaics on long-document question answering.
The goal of this research is to help language models remember new information and learn new tasks from examples more effectively. It focuses on improving how models use information during inference, rather than simply learning more knowledge during training. It is not yet a replacement for Transformers because its main advantages appear in specific memory and learning tasks, while its overall performance on standard benchmarks is similar and its efficiency at large scale is not explored.
📝 DeepLoop: Depth Scaling for Looped Transformers [paper] [* code**]* Training very deep Transformers can become unstable because small changes can build up as information passes through many layers. This becomes even harder when the same model components are used many times, because their updates can have a stronger combined effect.
Looped Transformers reuse the same layers several times instead of adding new ones. This gives the model more computation without adding many more parameters. DeepLoop shows that this repeated reuse changes how strongly updates can affect the model, so the scaling rule should be adjusted.
Residual connections are important because they help information and gradients flow through deep models and important for stability during training. Residual connections carry the current representation forward while the attention or MLP branch adds a new change. DeepLoop changes this balance by making the skip path stronger with α and starting the residual branch with smaller weights using β.
Their main contribution is to propose new scaling factors for the residual connection, instead of using the DeepNorm values designed for standard Transformers. In particular, they change the scaling exponent from 1/4 to 1/2 so the model stays more stable when the same layers are reused many times.
DeepLoop improves validation loss more as the number of loops increases. It also gives better average accuracy on most downstream settings, including a +3.25 point gain on ARC-AGI at the main voting setting. This suggests the new scaling becomes more useful when layer reuse is stronger.
I send out a monthly newsletter for NLP nerds. Consider subscribing if you like to stay up-to-date on the latest developments in Natural Language Processing. Read more and subscribe — join the cool kids club and sign up now!
What do you think of this newsletter? I would like to hear your feedback.
What parts were interesting? What sections did you not like? What was missing, and do you want to see more of it in future?
Please reach out to me at nlpiation@gmail.com. Month in 4 Papers (October 2026) was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.