Memorilla: Latent Semantic Memory for LLMs

A paper posted to arXiv on 15 October 2026 (revised 3 November 2026) by Vignesh Kothapalli introduces Memorilla, a memory system that condenses an entire document collection into 16 query-dependent 'memory tokens' for a frozen LLM. Memorilla beat baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering, and reached 84.33% accuracy on FactKG versus 80.43% for full-context inference while using 108x fewer tokens. In a TextWorld environment, Memorilla raised an agent's average success rate from 18.6% with full-history in-context learning to 24.2%.

Computer Science Machine Learning Submitted on 15 Oct 2026 v1 https://arxiv.org/abs/2610.21375v1 , last revised 3 Nov 2026 this version, v3 Title:Memorilla: Latent Semantic Memory for LLMs View PDF https://arxiv.org/pdf/2610.21375 HTML experimental https://arxiv.org/html/2610.21375v3 Abstract:Memory enables large language models LLMs to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen LLM, access to this accumulated knowledge is limited to what a memory system places in its context window. However, gathering and combining relevant information needed for each query while keeping the model's context short remains challenging for existing systems. Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Memorilla encodes documents such as user chats, book chapters once into reusable embeddings, then learns to retrieve and place query-specific information in the LLM's input embedding space. Using only 16 memory tokens, Memorilla achieves the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering. Notably, on FactKG, it reaches 84.33% accuracy versus 80.43% with full-context inference, while using 108x fewer tokens to represent the source documents. Furthermore, Memorilla can facilitate continual learning in agentic systems where the document count the agent's past turns increases throughout the trajectory. In a TextWorld environment, we show that Memorilla reduces the agent's tendency to wander between rooms it has already explored, and improves its average success rate from 18.6% with full-history in-context learning to 24.2%. Overall, Memorilla aids frozen LLMs in accumulating and using knowledge effectively across long-context interactions and takes a step towards building continually learning AI systems. Submission history From: Vignesh Kothapalli view email https://arxiv.org/show-email/bc7e86c1/2610.21375 Thu, 15 Oct 2026 06:55:55 UTC 3,660 KB \ v1\ https://arxiv.org/abs/2610.21375v1 Tue, 27 Oct 2026 04:28:12 UTC 3,660 KB \ v2\ https://arxiv.org/abs/2610.21375v2 v3 Tue, 3 Nov 2026 07:03:31 UTC 3,660 KB Additional Features References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender IArxiv Recommender What is IArxiv? https://iarxiv.org/about arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .