The sleep framework: A new way for LLMs to retain knowledge in continual learning

Google Research staff research scientist Ali Behrouz and VP and Google Fellow Vahab Mirrokni introduced a "sleep framework" for continual learning that alternates active interaction with offline memory consolidation, enabling large language models to retain long-context knowledge up to 10M tokens and improve complex reasoning while mitigating catastrophic forgetting. During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts memory blocks and updates its numerical weights using self-generated practice examples, within a fixed architecture and human-defined training rules. The framework has two stages, memory consolidation and dreaming, and the researchers ground it in a theory that models knowledge seeding as a structured, low-rank compression problem.

October 27, 2026 Ali Behrouz, Staff Research Scientist, and Vahab Mirrokni, VP and Google Fellow, Google Research We introduce a sleep phase in continual learning, enabling large language models to continuously integrate new facts and enhance reasoning while mitigating catastrophic forgetting or static knowledge cutoffs. Modern LLMs achieve impressive performance via pre-training and in-context learning https://ai.stanford.edu/blog/understanding-incontext/ ICL . However, once deployed, they remain largely static, operating with a fixed knowledge cutoff date that leaves them unable to acquire long-term capabilities or adapt to new information. While updating models with new data seems like the obvious solution, continual parameter updates often trigger catastrophic forgetting https://en.wikipedia.org/wiki/Catastrophic interference , a fundamental issue where learning new tasks severely degrades performance on previously learned ones. Alternatively, relying strictly on ICL limits knowledge retention to the immediate context window: once the context session ends, the acquired short-term information is forgotten. Inspired by how the human brain processes and stabilizes memory during rest, we introduce a sleep framework https://arxiv.org/pdf/2606.03979 in continual learning: a learning paradigm that equips large language models LLMs with a continuous periodic lifecycle, enabling them to continually learn, consolidate fragile short-term context into stable long-term parameters. With this framework, we demonstrate that alternating between active interaction and offline memory consolidation allows LLMs to continuously acquire new facts, retain long-context knowledge up to 10M tokens, and improve complex reasoning. Traditional machine learning relies on a strict separation between training and inference. In a continual learning framework, this distinction becomes artificial; an intelligent agent must constantly learn, adapt, and process information. Under the sleep paradigm, an LLM operates across two primary states: While terms like "sleep," "dreaming," and "self-improvement" borrow from neuroscience, our framework operates entirely within a fixed model architecture and human-defined training rules. During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts MoE memory blocks and updates its numerical weights using self-generated practice examples. This automated post-training cycle helps a deployed model absorb new facts and sharpen its reasoning accuracy while keeping its core codebase and design unchanged. The sleep paradigm addresses catastrophic forgetting and continuous adaptation through two stages: memory consolidation and dreaming. In the human brain, non-rapid eye movement https://en.wikipedia.org/wiki/Non-rapid eye movement sleep NREM slow-wave sleep consolidates fragile memories into long-term structures via abstraction and distillation. Our proposed sleep framework mirrors this natural phenomenon through two key technical mechanisms: During REM sleep https://www.thensf.org/what-is-rem-sleep/ , the human brain actively simulates scenarios, consolidates emotional/semantic connections, and explores novel neural pathways. Mirroring this, during the “dreaming stage”, the sleep framework implements an autonomous self-improvement loop: Beyond the empirical results, our sleep paradigm is grounded in a theory that clarifies why its two stages memory consolidation and dreaming can support continual learning without simply trading new knowledge for old capabilities. For memory consolidation, the analysis shows that knowledge seeding can be understood as a structured, low-rank compression problem. The model is not asked to copy every detail of its short-term traces into its parameters. Instead, it identifies a compact set of directions that best preserve both the model’s own on-policy behavior and the semantic qualities rewarded by the learning objective. This perspective explains the complementary roles of the two signals: on-policy distillation aligns consolidation with the trajectories the model will actually produce at inference time, reducing the mismatch that can arise when training only on teacher-generated distributions; meanwhile, the semantic reward changes what is worth retaining, prioritizing directions that preserve meaning and task-relevant behavior rather than merely matching token probabilities. Parameter expansion then provides the capacity needed to write these selected memories without overwriting previously consolidated knowledge. For dreaming, the theory characterizes random expert activation and reward-based sample selection as mechanisms for improving the geometry of learning. Injecting different MoEs broadens the set of gradient directions explored during synthetic generation, exposing the model to a richer collection of updates than those induced by its ordinary wake-time data alone. Gradient-based filtering then emphasizes the synthetic experiences that carry the most useful learning signal, strengthening informative directions while preserving the structure of the underlying gradient space. Together, these steps yield a more balanced optimization landscape: learning is less dominated by a small number of directions, gradients are better conditioned, and the model can make more stable progress when refining its capabilities. In this view, dreaming is not simply self-generated data augmentation; it is a deliberate process for diversifying, selecting, and consolidating the updates through which the model improves itself. For continual translation https://arxiv.org/abs/2512.24695 , in unseen languages, such as Manchu and Kalamang, the sleep framework maintained a near-zero performance drop in multi-stage evaluations, whereas standard baselines like ICL, Cartridges https://arxiv.org/abs/2506.06266 , and supervised fine-tuning https://cameronrwolfe.substack.com/p/understanding-and-using-supervised SFT suffered severe catastrophic forgetting during sequential learning. In long-context understanding on the BABILong https://arxiv.org/abs/2406.10149 benchmark, our sleep method demonstrated near-perfect scaling up to 10M tokens, outperforming strong baselines like GPT-4 http://openai.com/index/gpt-4-research/ , and Titans https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/ by using sleep-time consolidation to compress long-context information directly into parametric memory. For factual knowledge incorporation evaluated on SQuAD https://deepeval.com/docs/benchmarks-squad and continued pre-training https://docs.aws.amazon.com/nova/latest/userguide/nova-cpt.html CPT , our sleep method achieved 48.9% on single-task and 46.2% on CPT evaluations, outperforming baseline methods including the base model, SFT, and Self-Adapting Language Models https://arxiv.org/abs/2506.10943 SEAL by successfully integrating new factual information without degrading existing model representations. Finally, on mathematical reasoning benchmarks including AIME 24/25 https://artificialanalysis.ai/evaluations/aime-2025 and HMMT 25 https://intuitionlabs.ai/articles/hmmt25-ai-benchmark-explained , sleep-enhanced base-model achieved accuracy scores of 79.2%, 69.0%, and 46.1%, respectively. This outperformed baselines like SFT, group relative policy optimization GRPO , and on-policy self-distillation OPSD , showing that pairing memory consolidation with imitation learning drives superior reasoning performance. Our sleep paradigm provides a path toward autonomous, continually learning AI systems. By moving beyond static pre-training and fragile context windows, alternating between active learning and offline rest empowers LLMs to retain short-term memories, abstract new skills into long-term parameter structures, and continuously self-improve without catastrophic forgetting. This research was conducted by Ali Behrouz Google and Cornell University , Farnoosh Hashemi Cornell University , Adel Javanmard Google and University of Southern California , and Vahab Mirrokni. We thank Yuan Deng and Corinna Cortes for reviewing the work and their valuable suggestions. We also thank Mark Simborg, Kimberly Schwede, and Tim Herrmann for their help in crafting this blog post.