Memorilla: Stanford’s Latent Memory — 16 Tokens vs Full Context on Frozen Qwen3-8B

Memorilla is a proposal for latent semantic memory over large collections without stuffing transcripts into the context window. The authors, led by researchers at Stanford including Jure Leskovec, describe the method in arXiv:2610.21375 and on memory.stanford.edu. A trainable memory module reads every stored embedding for each query and feeds a frozen Qwen3-8B decoder sixteen learned memory tokens, which are soft vectors in the model’s input embedding space, instead of raw text.

This post states what the authors claim, how the mechanism works at a high level, where it differs from retrieval-augmented generation, and which limits they flag. It does not reproduce the paper’s architecture appendices.

TL;DR #

How Memorilla differs from chat summaries and Mem0-style logs #

Industry stacks often choose among three patterns:

• Summaries stored as plain text: cheap to interpret, brittle to selection.

• Structured memories in frameworks such as Mem0. Our Hy-Memory writer for Mem0 write-up shows how vendors now specialize write paths.

• Agent-facing memory systems that retrieve and write notes across sessions, such as Hindsight from Vectorize andDatabricks MemCurrent . These operate at the text or record level, so they differ from Memorilla's soft-token readout.

• Latent caches such as MemT-1.0 that drop tokens after compression. Memorilla sits closest to the latent branch, but with a different objective. It attends across the whole embedding matrix on every query rather than picking top-k chunks first. The trade-off is computational: a full-matrix attention read scales with collection size, while context length scales with the tokens you paste.

Architecture in plain language #

The paper’s encoding stage is ordinary dense embedding:

1. Split a document collection into chunks.
2. Run each chunk through a frozen document encoder once, producing a row in a matrix Z that you persist.

At query time:

1. Encode the query to a vector.
2. Initialize K learnable slots, update them with a cross-attention step against the query vector, then run several cross-attention blocks against all rows of Z. This is a Perceiver-style latent readout.
3. Project the slot vectors into the frozen decoder’s embedding width and prepend them before the ordinary text embeddings of the user question.

Training uses a standard next-token loss on answer spans. The decoder weights stay fixed.

That framing matters for builders. The project page says the method requires open model weights and a memory module trained for each LLM, so you cannot bolt Memorilla onto a closed API that hides the input embedding layer.

Compute characteristics you should plan for

The advantage Memorilla chases is decoder token count: 16 memory slots plus the question text instead of thousands of document tokens. The cost moves upstream:

• Storage: every chunk in Z needs a vector at encoder width.
• Per-query work: each cross-attention block touches all N rows. In the naive picture that scales with N, not with a small k as in approximate-nearest-neighbor top-k RAG. This is explainx.ai’s reading of the design, not a measured result.
• Encoder amortization: new documents need one extra forward pass, the same as adding to a dense retrieval index.

For small N, such as personal assistant histories or a few hundred onboarding documents, that trade may win. For web-scale N you would likely pre-filter the store unless the hardware budget is large. The latency claim on the project page is 0.74 s versus 19.9 s with 512 documents on one H100-80GB. It compares Memorilla against full-context answering and is a single-GPU, batch-size-1 setup. It is not a substitute for testing on your own serving layout.

Very long scroll with a small green marker, illustrating stretching context versus compact latent memory for long corpora

Training recipe and baselines #

The paper describes three supervised stages:

1. Paraphrase reconstruction on about 1.98 million English Wikipedia examples, so the memory slots learn to preserve content under compression.
2. A multi-task mixture of about 1.34 million examples from ten reading-comprehension, open-domain QA and summarization datasets, to expose diverse formats.
3. Fine-tuning on a mixture of the training splits of the evaluation benchmarks.

For the 8B decoder the paper reports 194,123,776 trainable parameters in the memory module and projection, and 190,190,080 for the 4B version. That is small next to the decoder, but not free at serving scale. Baselines use frozen Qwen3 models: cosine-similarity RAG, Mem0, MemoRAG, xRAG, LLMLingua-2 prompt compression, and full context. Where the baseline input exceeds the native length, the full-context run uses YaRN. This controls for generator strength but not for engineering maturity. Mem0 and MemoRAG are active products with tuning knobs outside the paper’s grid.

What the authors report on benchmarks #

The project page highlights these numbers from the Qwen3-8B setting:

• A mean score of 49.68 across eight benchmarks, the highest of all methods tested, with 394× fewer document tokens than full context on average. The abstract says Memorilla beats baselines including Mem0 and MemoRAG on five of the eight.
• FactKG accuracy of 84.33% versus 80.43% for full context, with 108× fewer document tokens. RAG-50 scored 78.4%.
• TriviaQA exact match of 57.0 versus 52.60 for full context, with 743× fewer document tokens.
• PersonaMem-v2 short-answer accuracy of 44.60% versus 30.90% for the best baseline (xRAG) and 27.92% for full context, with 1,974× fewer document tokens.
• NarrativeQA: 16 memory tokens in place of 67,385 document tokens.
• Latency with 512 stored documents: 0.74 s versus 19.9 s for full context (27× lower).

PubMedQA is a regression: 59.00 accuracy versus 67.00 for full context and 69.00 for the best baseline, LLMLingua-2. Short biomedical passages may still favor verbatim text; that is explainx.ai’s reading, not an explanation the authors confirm.

Readers comparing to frontier APIs should remember that none of these numbers come from GPT-class or Claude-family decoders. The paper tests only Qwen encoder and decoder pairs.

TextWorld: memory for agents, not magic autonomy #

The TextWorld experiments switch to Qwen3-4B with eight memory tokens. The paper reports average success of 24.2% versus 18.6% for full-history in-context prompting. The share of held-out games completed within three attempts rose from 25.0% to 37.5%. The abstract also says the agent revisits already-explored rooms less often.

Those lifts are real on their suite but small in absolute terms. Treat this as a signal that agents benefit from state beyond last-turn recaps, not as a solved benchmark. The authors note the results cover one game suite with episodes of at most 50 turns.

If your team ships agents, pair this with our memory tax explainer: naive RAG still beats elaborate memory on some tasks when measurement is blunt. Green thread linking an answer to stacked source pages, comparing latent memory compression with classical retrieval text

Memorilla versus RAG in one paragraph #

RAG gives the decoder exact strings it can copy, which is great for proper nouns. Memorilla never exposes those strings unless the decoder reconstructs them from latent hints. That suits cases where connections span many weakly matching passages, and it is worse when the answer is a verbatim span.

Honest limitations (from the authors) #

The project page lists these constraints:

1. Detail loss. Exact names can fade when only 16 latent slots carry information.
2. Per-model training. The module needs open model weights and must be trained for each LLM, because the projection binds it to one decoder’s embedding space.
3. Scope. Tests used only Qwen encoder and decoder pairs, on text. Other model families, modalities and domains are untested.
4. Agent scope. TextWorld results cover one game suite with episodes of at most 50 turns.

None of that invalidates the core idea. It bounds where to try it first: personalization with diffuse cues, or long corpora where RAG-50 misses. It is a poor fit for legal-citation bots that need byte-level faithfulness.

What developers can try today #

The project page links code at github.com/snap-stanford/memorilla and weights on Hugging Face. The paper is also cited as a COLM 2026 workshop paper (Context Beyond the Window: Persistent Knowledge in Language Models). Check the repository license and checkpoint list before planning production forks.

If you experiment:

1. Instrument attention entropy over the store. Flat attention suggests the module is not selecting anything useful.
2. Ablate K. The paper uses 16 tokens; your corpus may need more or fewer.
3. Consider a cheap BM25 pre-filter if the number of embeddings heads into the millions. Memorilla reads the whole store per query, so cost grows with N.

Related reading #

Claims and tables above follow arXiv:2610.21375 and memory.stanford.edu as of November 6, 2026. Independent reproductions were not available at publication time.