Stanford Memorilla: 16 Tokens Beat Full Context and RAG Stanford's SNAP group open-sourced Memorilla on November 14, a 194-million-parameter memory module that compresses a document collection into 16 latent tokens and injects them into a frozen LLM, scoring 84.33% on the FactKG factual reasoning benchmark versus 80.43% for full context and 80.38% for MemoRAG. Memorilla uses 108 times fewer document tokens and responds in 0.74 seconds versus 19.9 seconds for full-context prompting at 512 documents on an H100, with the bridge module trained for 19 hours on 4 H100s while the Qwen3-Embedding-4B encoder and Qwen3-8B LLM stay frozen. On PersonaMem-v2, Memorilla reaches 44.60% short-answer accuracy across 107 historical conversations versus 30.90% for the best baseline, xRAG, though the authors document a failure mode in verbatim retrieval of specific facts. Stanford’s SNAP group open-sourced Memorilla https://github.com/snap-stanford/memorilla on November 14, a 194-million-parameter memory module that compresses an entire document collection into 16 latent tokens, injects them into a frozen LLM, and outperforms both RAG and full-context stuffing on most benchmarks. It runs 27 times faster than full-context prompting at 512 documents, uses 108 times fewer tokens, and still posts higher accuracy. The code is Apache 2.0 https://arxiv.org/abs/2610.21375 . The Problem With How LLMs Remember Things Today The two dominant patterns — shove everything into the context window, or retrieve-then-augment — both have serious failure modes that teams tend to discover in production rather than in demos. Full context is expensive and dumb. It treats every token equally, pays for all of them, and still gets distracted by irrelevant content. At 512 documents, a full-context response takes 19.9 seconds on an H100. More context doesn’t mean better attention — it means more noise. RAG starts strong but degrades. On single-hop factual questions it works fine. When queries require composing multiple facts, accuracy collapses: on a compositional multi-claim benchmark, RAG drops from 78.7% to 51.8% as the fact count grows. The retriever doesn’t know what the LLM will need — it just guesses based on semantic similarity. The core issue is that neither approach actually learns to remember. They retrieve or repeat. Memorilla does something different. How It Works Memorilla’s architecture is cleaner than it sounds. A frozen encoder Qwen3-Embedding-4B processes your documents once and stores the representations. That’s the encode-and-cache step — the encoder never runs again for the same documents. At query time, a 194-million-parameter bridge module — the only part that’s trained — reads the incoming question and those stored embeddings. It produces 16 memory vectors and injects them directly into the frozen LLM’s Qwen3-8B input embedding space. The LLM never sees the raw documents. It only sees those 16 tokens. What’s trained is just the bridge: 19 hours on 4 H100s, using standard answer-token loss. The encoder and the LLM stay frozen throughout. That means you’re not fine-tuning a large model — you’re training a small translator that learns to pack what matters into 16 slots. Memorilla treats memory as a modality for LLMs. Jure Leskovec, Stanford SNAP PI The Benchmark Numbers On FactKG, a factual reasoning benchmark requiring multi-hop knowledge retrieval, Memorilla scores 84.33% accuracy. Full context gets 80.43%. MemoRAG gets 80.38%. Memorilla uses 108 times fewer document tokens and responds in 0.74 seconds versus 19.9 seconds for full context at 512 documents. The personalization results are more striking. On PersonaMem-v2 — where an agent recalls context across 107 historical conversations totaling about 32,000 tokens — Memorilla achieves 44.60% short-answer accuracy. The best baseline, xRAG, gets 30.90%. Compressing 107 conversations into 16 latent tokens while improving recall by nearly 50% is the result worth paying attention to. In TextWorld navigation, where a frozen agent explores up to 65 rooms to complete a quest, compressing the growing trajectory history into 8 memory tokens improved task completion from 18.6% to 24.2%. Games completed within three attempts jumped from 25% to 37.5%. Agents stopped wandering rooms they had already explored. | Method | FactKG Accuracy | Token Compression | Latency 512 docs | |---|---|---|---| | Full Context | 80.43% | 1x | 19.9s | | MemoRAG | 80.38% | 36x fewer | ~15s | | xRAG | 78.98% | — | — | | Memorilla | 84.33% | 108x fewer | 0.74s | Where It Still Loses Memorilla has a documented failure mode: specific-fact verbatim retrieval. If a user asks “what is my sister’s birthday?” and that detail lives 50 conversations back, Memorilla can fabricate rather than admit it doesn’t know. RAG, which retrieves the actual text, still wins here. The researchers’ recommendation is to run Memorilla alongside RAG — Memorilla for semantic synthesis and compositional reasoning, RAG for exact-fact lookup. There’s also a practical constraint: Memorilla requires open model weights. Closed API models like GPT-4o or Claude are out. And the 19-hour training time per task means it’s not a drop-in today. Getting Started The code is at snap-stanford/memorilla https://github.com/snap-stanford/memorilla on GitHub under Apache 2.0. Pre-trained checkpoints for Qwen3-8B are on Hugging Face https://huggingface.co/memorilla . The project page https://memory.stanford.edu/ has interactive demos and the full benchmark breakdown. If you’re building agents that accumulate long histories, or personalization systems that need to recall across sessions, these numbers are worth testing against your current stack. The context window arms race has diminishing returns. Learned memory is a sharper tool for the retrieval tasks that actually matter at scale.