{"slug": "zero-forgetting-in-llms-we-benchmarked-null-space-projection-on-smollm-and-hit", "title": "Zero Forgetting in LLMs? We Benchmarked Null-Space Projection on SmolLM and Hit the Plasticity Wall", "summary": "A developer implemented and benchmarked Wake-Sleep Null-Space Consolidation (WS-NSC), a continual-learning method that projects LoRA weight updates onto the orthogonal null-space of prior activations, on SmolLM-135M across a four-stage sequential task setup. The approach eliminated backward forgetting (down to -0.39% versus +5.83% for direct merging) but cut downstream plasticity retention to 37.7% at a 0.90 variance threshold and 24.4% at 0.95, which the author attributes to a mathematical plasticity penalty. The code is open-sourced and the benchmark data registered under DOI 10.5281/zenodo.23763908.", "body_md": "Why orthogonal projections eliminate catastrophic forgetting in autoregressive Transformers—and the mathematical reason behind the 62% downstream plasticity penalty.\n\nFine-tuning Large Language Models sequentially without retraining from scratch is notoriously broken. When you adapt an LLM to **Task B** after **Task A**, it systematically overwrites the weight configurations that enabled performance on Task A. This is **Catastrophic Forgetting**.\n\nWhile Parameter-Efficient Fine-Tuning (PEFT) techniques like **LoRA** (Low-Rank Adaptation) isolate task-specific updates into small low-rank matrices:\n\n$$\\Delta W = \\frac{\\alpha}{r} B A$$\n\na critical bottleneck arises during deployment: **How do you consolidate multiple sequential adapters into a single base model without incurring catastrophic interference?**\n\nDirectly merging delta weights:\n\n$$W_{\\text{base}} \\leftarrow W_{\\text{base}} + \\Delta W$$\n\ndestroys historical task performance.\n\nTo evaluate this frontier, we implemented and benchmarked **Wake-Sleep Null-Space Consolidation (WS-NSC)** on **SmolLM-135M**. The code is open-sourced, and the complete benchmark data is permanently registered under DOI **[10.5281/zenodo.23763908](https://doi.org/10.5281/zenodo.23763908)**.\n\nHere is the empirical reality of what happens when you mathematically force parameter updates into the orthogonal null-space of prior activations.\n\nThe framework operates in a biologically inspired two-phase regime:\n\n```\n[Wake Phase: Online LoRA Training]\n            │\n            ▼\n[Activation Manifold Collection: C = Σ (x^T x) / M]\n            │\n            ▼\n[Sleep Phase: Null-Space Orthogonal Projection]\n            │\n            ▼\n[Consolidation: W_base ← W_base + ΔW @ P_null]\n```\n\nThe model freezes base parameters $W_0$ and trains a standard LoRA adapter $(A_t, B_t)$ on the active task stream $\\mathcal{D}_t$.\n\nForward hooks register linear input activations $x \\in \\mathbb{R}^{B \\times L \\times d_{\\text{in}}}$ across reference tokens and historical tasks. We compute the uncentered empirical second-moment covariance matrix:\n\n$$C = \\frac{1}{M} \\sum_{i=1}^M x_i^\\top x_i \\in \\mathbb{R}^{d_{\\text{in}} \\times d_{\\text{in}}}$$\n\nWe extract eigenvectors $U$ and eigenvalues $\\Lambda$ such that $C = U \\Lambda U^\\top$. We retain the top-$k$ principal components accounting for a variance threshold $\\rho \\in [0.90, 0.95]$:\n\n$$\\frac{\\sum_{i=1}^k \\lambda_i}{\\sum_{j=1}^d \\lambda_j} \\ge \\rho$$\n\nThe parameter update $\\Delta W$ is then projected onto the exact null-space of $U_k$:\n\n$$P_{\\text{null}} = I - U_k U_k^\\top$$\n\n$$\\Delta W_{\\text{safe}} = \\Delta W \\cdot P_{\\text{null}}$$\n\nBecause $x U_k \\approx x$ for historical activations, projecting into $P_{\\text{null}}$ ensures:\n\n$$\\Delta W_{\\text{safe}} \\cdot x^\\top \\approx 0$$\n\nTheoretical guarantee: **Prior task forward passes experience zero functional shift.**\n\nWe configured a 4-stage sequential continual learning setup:\n\nAll experiments were executed on an A100 GPU using PyTorch AMP across all 7 projection matrices (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`).\n\n| Consolidation Strategy | Wake Gain (%) | Post-Sleep Gain (%) | Plasticity Retention | Forgetting (nats) | Relative Forgetting (%) | Anchor Drift $\\Delta$ | Protected Rank ($k/d$) | Update Norm Kept | \n|---|---|---|---|---|---|---|---|---|\n| **Direct-Merge** | **16.35%** | **16.32%** | **99.8%** | $+0.103$ | $+5.833\\%$ | $+0.057$ | $0.000$ | $100.0\\%$ | \n| **WS-NSC ($\\rho=0.90$)** | 15.29% | 5.76% | **37.7%** | $\\mathbf{-0.007}$ | $\\mathbf{-0.390\\%}$ | $\\mathbf{-0.005}$ | $36.2\\%$ | $73.9\\%$ | \n| **WS-NSC ($\\rho=0.95$)** | 15.33% | 3.74% | **24.4%** | $\\mathbf{-0.007}$ | $-0.387\\%$ | $-0.002$ | $54.5\\%$ | $61.6\\%$ | \n\nIn direct merging, catastrophic forgetting is glaring: backward tasks degrade by $+5.83\\%$, and the general-domain anchor drifts by $+0.057$ nats.\n\nUnder WS-NSC ($\\rho=0.90$), backward forgetting drops to **$-0.39\\%$** and anchor drift drops to **$-0.005$ nats**. Historical knowledge is preserved without storing or replaying raw pretraining datasets.\n\nWhile stability is solved, it incurs an acute functional cost. Under $\\rho=0.90$, **$36.2\\%$ of the linear subspace is locked down**, causing post-sleep task performance to retain only **$37.7\\%$** of what was acquired during the wake phase.\n\nPushing protection to $\\rho=0.95$ locks **$54.5\\%$** of the rank, collapsing retained gains down to **$24.4\\%$**. In deep nonlinear architectures, hard orthogonal projection chokes forward model capacity.\n\nA common assumption in model merging is that preserving weight magnitude preserves task utility.\n\nOur data disproves this: **retaining $73.9\\%$ of parameter update norm resulted in retaining only $37.7\\%$ of task gain.** Spectral truncation clips the highest-energy activation vectors needed for downstream task execution.\n\nThe full codebase, automated plotting pipelines, and raw execution logs are open for external verification:\n\nTo reproduce the benchmark locally:\n\n```\ngit clone https://github.com/AkshatRaj00/ws-nsc.git\ncd ws-nsc\npip install -r requirements.txt\n\n# Run empirical sequential evaluation\npython src/ws_nsc.py --model HuggingFaceTB/SmolLM-135M --amp --batch-size 4 --eval-batch-size 4 --var-retention 0.90 0.95\n\n# Generate loss trajectories & Pareto frontier plots\npython src/plot_results.py\n@article{raj2026wsnsc,\n  title={Wake-Sleep Null-Space Consolidation: Quantifying the Stability-Plasticity Frontier in Sequential LLM Adaptation},\n  author={Raj, Akshat},\n  year={2026},\n  publisher={Zenodo},\n  doi={10.5281/zenodo.23763908},\n  url={https://doi.org/10.5281/zenodo.23763908}\n}\n```\n\n`machinelearning` `datascience` `ai` `deeplearning` `python`", "url": "https://wpnews.pro/news/zero-forgetting-in-llms-we-benchmarked-null-space-projection-on-smollm-and-hit", "canonical_source": "https://dev.to/akshatraj00/zero-forgetting-in-llms-we-benchmarked-null-space-projection-on-smollm-and-hit-the-plasticity-wall-4n7a", "published_at": "2026-10-29 12:06:32+00:00", "updated_at": "2026-10-29 12:17:35.012735+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "mlops"], "entities": ["SmolLM-135M", "PyTorch", "Zenodo", "A100"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/zero-forgetting-in-llms-we-benchmarked-null-space-projection-on-smollm-and-hit", "markdown": "https://wpnews.pro/news/zero-forgetting-in-llms-we-benchmarked-null-space-projection-on-smollm-and-hit.md", "text": "https://wpnews.pro/news/zero-forgetting-in-llms-we-benchmarked-null-space-projection-on-smollm-and-hit.txt", "jsonld": "https://wpnews.pro/news/zero-forgetting-in-llms-we-benchmarked-null-space-projection-on-smollm-and-hit.jsonld"}}