cd /news/artificial-intelligence/stanford-memorilla-16-tokens-beat-fu… · home › topics › artificial-intelligence › article
[ARTICLE · art-170789] src=byteiota.com ↗ pub= topic=artificial-intelligence verified=true sentiment=↑ positive

Stanford Memorilla: 16 Tokens Beat Full Context and RAG

Stanford's SNAP group open-sourced Memorilla on November 14, a 194-million-parameter memory module that compresses a document collection into 16 latent tokens and injects them into a frozen LLM, scoring 84.33% on the FactKG factual reasoning benchmark versus 80.43% for full context and 80.38% for MemoRAG. Memorilla uses 108 times fewer document tokens and responds in 0.74 seconds versus 19.9 seconds for full-context prompting at 512 documents on an H100, with the bridge module trained for 19 hours on 4 H100s while the Qwen3-Embedding-4B encoder and Qwen3-8B LLM stay frozen. On PersonaMem-v2, Memorilla reaches 44.60% short-answer accuracy across 107 historical conversations versus 30.90% for the best baseline, xRAG, though the authors document a failure mode in verbatim retrieval of specific facts.

read4 min views3 publishedNov 16, 2026
Stanford Memorilla: 16 Tokens Beat Full Context and RAG
Image: Byteiota (auto-discovered)

Stanford’s SNAP group open-sourced Memorilla on November 14, a 194-million-parameter memory module that compresses an entire document collection into 16 latent tokens, injects them into a frozen LLM, and outperforms both RAG and full-context stuffing on most benchmarks. It runs 27 times faster than full-context prompting at 512 documents, uses 108 times fewer tokens, and still posts higher accuracy. The code is Apache 2.0.

The Problem With How LLMs Remember Things Today #

The two dominant patterns — shove everything into the context window, or retrieve-then-augment — both have serious failure modes that teams tend to discover in production rather than in demos.

Full context is expensive and dumb. It treats every token equally, pays for all of them, and still gets distracted by irrelevant content. At 512 documents, a full-context response takes 19.9 seconds on an H100. More context doesn’t mean better attention — it means more noise.

RAG starts strong but degrades. On single-hop factual questions it works fine. When queries require composing multiple facts, accuracy collapses: on a compositional multi-claim benchmark, RAG drops from 78.7% to 51.8% as the fact count grows. The retriever doesn’t know what the LLM will need — it just guesses based on semantic similarity.

The core issue is that neither approach actually learns to remember. They retrieve or repeat. Memorilla does something different.

How It Works #

Memorilla’s architecture is cleaner than it sounds. A frozen encoder (Qwen3-Embedding-4B) processes your documents once and stores the representations. That’s the encode-and-cache step — the encoder never runs again for the same documents.

At query time, a 194-million-parameter bridge module — the only part that’s trained — reads the incoming question and those stored embeddings. It produces 16 memory vectors and injects them directly into the frozen LLM’s (Qwen3-8B) input embedding space. The LLM never sees the raw documents. It only sees those 16 tokens.

What’s trained is just the bridge: 19 hours on 4 H100s, using standard answer-token loss. The encoder and the LLM stay frozen throughout. That means you’re not fine-tuning a large model — you’re training a small translator that learns to pack what matters into 16 slots.

Memorilla treats memory as a modality for LLMs.

Jure Leskovec, Stanford SNAP PI

The Benchmark Numbers #

On FactKG, a factual reasoning benchmark requiring multi-hop knowledge retrieval, Memorilla scores 84.33% accuracy. Full context gets 80.43%. MemoRAG gets 80.38%. Memorilla uses 108 times fewer document tokens and responds in 0.74 seconds versus 19.9 seconds for full context at 512 documents.

The personalization results are more striking. On PersonaMem-v2 — where an agent recalls context across 107 historical conversations totaling about 32,000 tokens — Memorilla achieves 44.60% short-answer accuracy. The best baseline, xRAG, gets 30.90%. Compressing 107 conversations into 16 latent tokens while improving recall by nearly 50% is the result worth paying attention to.

In TextWorld navigation, where a frozen agent explores up to 65 rooms to complete a quest, compressing the growing trajectory history into 8 memory tokens improved task completion from 18.6% to 24.2%. Games completed within three attempts jumped from 25% to 37.5%. Agents stopped wandering rooms they had already explored.

Method FactKG Accuracy Token Compression Latency (512 docs)
Full Context 80.43% 1x 19.9s
MemoRAG 80.38% 36x fewer ~15s
xRAG 78.98% — —
Memorilla 84.33% 108x fewer 0.74s

Where It Still Loses #

Memorilla has a documented failure mode: specific-fact verbatim retrieval. If a user asks “what is my sister’s birthday?” and that detail lives 50 conversations back, Memorilla can fabricate rather than admit it doesn’t know. RAG, which retrieves the actual text, still wins here.

The researchers’ recommendation is to run Memorilla alongside RAG — Memorilla for semantic synthesis and compositional reasoning, RAG for exact-fact lookup. There’s also a practical constraint: Memorilla requires open model weights. Closed API models like GPT-4o or Claude are out. And the 19-hour training time per task means it’s not a drop-in today.

Getting Started #

The code is at snap-stanford/memorilla on GitHub under Apache 2.0. Pre-trained checkpoints for Qwen3-8B are on Hugging Face. The project page has interactive demos and the full benchmark breakdown.

If you’re building agents that accumulate long histories, or personalization systems that need to recall across sessions, these numbers are worth testing against your current stack. The context window arms race has diminishing returns. Learned memory is a sharper tool for the retrieval tasks that actually matter at scale.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @stanford snap 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/stanford-memorilla-1…] indexed:0 read:4min 2026-11-16 · —