Zero Forgetting in LLMs? We Benchmarked Null-Space Projection on SmolLM and Hit the Plasticity Wall

A developer implemented and benchmarked Wake-Sleep Null-Space Consolidation (WS-NSC), a continual-learning method that projects LoRA weight updates onto the orthogonal null-space of prior activations, on SmolLM-135M across a four-stage sequential task setup. The approach eliminated backward forgetting (down to -0.39% versus +5.83% for direct merging) but cut downstream plasticity retention to 37.7% at a 0.90 variance threshold and 24.4% at 0.95, which the author attributes to a mathematical plasticity penalty. The code is open-sourced and the benchmark data registered under DOI 10.5281/zenodo.23763908.

Why orthogonal projections eliminate catastrophic forgetting in autoregressive Transformers—and the mathematical reason behind the 62% downstream plasticity penalty. Fine-tuning Large Language Models sequentially without retraining from scratch is notoriously broken. When you adapt an LLM to Task B after Task A , it systematically overwrites the weight configurations that enabled performance on Task A. This is Catastrophic Forgetting . While Parameter-Efficient Fine-Tuning PEFT techniques like LoRA Low-Rank Adaptation isolate task-specific updates into small low-rank matrices: $$\Delta W = \frac{\alpha}{r} B A$$ a critical bottleneck arises during deployment: How do you consolidate multiple sequential adapters into a single base model without incurring catastrophic interference? Directly merging delta weights: $$W {\text{base}} \leftarrow W {\text{base}} + \Delta W$$ destroys historical task performance. To evaluate this frontier, we implemented and benchmarked Wake-Sleep Null-Space Consolidation WS-NSC on SmolLM-135M . The code is open-sourced, and the complete benchmark data is permanently registered under DOI 10.5281/zenodo.23763908 https://doi.org/10.5281/zenodo.23763908 . Here is the empirical reality of what happens when you mathematically force parameter updates into the orthogonal null-space of prior activations. The framework operates in a biologically inspired two-phase regime: Wake Phase: Online LoRA Training │ ▼ Activation Manifold Collection: C = Σ x^T x / M │ ▼ Sleep Phase: Null-Space Orthogonal Projection │ ▼ Consolidation: W base ← W base + ΔW @ P null The model freezes base parameters $W 0$ and trains a standard LoRA adapter $ A t, B t $ on the active task stream $\mathcal{D} t$. Forward hooks register linear input activations $x \in \mathbb{R}^{B \times L \times d {\text{in}}}$ across reference tokens and historical tasks. We compute the uncentered empirical second-moment covariance matrix: $$C = \frac{1}{M} \sum {i=1}^M x i^\top x i \in \mathbb{R}^{d {\text{in}} \times d {\text{in}}}$$ We extract eigenvectors $U$ and eigenvalues $\Lambda$ such that $C = U \Lambda U^\top$. We retain the top-$k$ principal components accounting for a variance threshold $\rho \in 0.90, 0.95 $: $$\frac{\sum {i=1}^k \lambda i}{\sum {j=1}^d \lambda j} \ge \rho$$ The parameter update $\Delta W$ is then projected onto the exact null-space of $U k$: $$P {\text{null}} = I - U k U k^\top$$ $$\Delta W {\text{safe}} = \Delta W \cdot P {\text{null}}$$ Because $x U k \approx x$ for historical activations, projecting into $P {\text{null}}$ ensures: $$\Delta W {\text{safe}} \cdot x^\top \approx 0$$ Theoretical guarantee: Prior task forward passes experience zero functional shift. We configured a 4-stage sequential continual learning setup: All experiments were executed on an A100 GPU using PyTorch AMP across all 7 projection matrices q proj , k proj , v proj , o proj , gate proj , up proj , down proj . | Consolidation Strategy | Wake Gain % | Post-Sleep Gain % | Plasticity Retention | Forgetting nats | Relative Forgetting % | Anchor Drift $\Delta$ | Protected Rank $k/d$ | Update Norm Kept | |---|---|---|---|---|---|---|---|---| | Direct-Merge | 16.35% | 16.32% | 99.8% | $+0.103$ | $+5.833\%$ | $+0.057$ | $0.000$ | $100.0\%$ | | WS-NSC $\rho=0.90$ | 15.29% | 5.76% | 37.7% | $\mathbf{-0.007}$ | $\mathbf{-0.390\%}$ | $\mathbf{-0.005}$ | $36.2\%$ | $73.9\%$ | | WS-NSC $\rho=0.95$ | 15.33% | 3.74% | 24.4% | $\mathbf{-0.007}$ | $-0.387\%$ | $-0.002$ | $54.5\%$ | $61.6\%$ | In direct merging, catastrophic forgetting is glaring: backward tasks degrade by $+5.83\%$, and the general-domain anchor drifts by $+0.057$ nats. Under WS-NSC $\rho=0.90$ , backward forgetting drops to $-0.39\%$ and anchor drift drops to $-0.005$ nats . Historical knowledge is preserved without storing or replaying raw pretraining datasets. While stability is solved, it incurs an acute functional cost. Under $\rho=0.90$, $36.2\%$ of the linear subspace is locked down , causing post-sleep task performance to retain only $37.7\%$ of what was acquired during the wake phase. Pushing protection to $\rho=0.95$ locks $54.5\%$ of the rank, collapsing retained gains down to $24.4\%$ . In deep nonlinear architectures, hard orthogonal projection chokes forward model capacity. A common assumption in model merging is that preserving weight magnitude preserves task utility. Our data disproves this: retaining $73.9\%$ of parameter update norm resulted in retaining only $37.7\%$ of task gain. Spectral truncation clips the highest-energy activation vectors needed for downstream task execution. The full codebase, automated plotting pipelines, and raw execution logs are open for external verification: To reproduce the benchmark locally: git clone https://github.com/AkshatRaj00/ws-nsc.git cd ws-nsc pip install -r requirements.txt Run empirical sequential evaluation python src/ws nsc.py --model HuggingFaceTB/SmolLM-135M --amp --batch-size 4 --eval-batch-size 4 --var-retention 0.90 0.95 Generate loss trajectories & Pareto frontier plots python src/plot results.py @article{raj2026wsnsc, title={Wake-Sleep Null-Space Consolidation: Quantifying the Stability-Plasticity Frontier in Sequential LLM Adaptation}, author={Raj, Akshat}, year={2026}, publisher={Zenodo}, doi={10.5281/zenodo.23763908}, url={https://doi.org/10.5281/zenodo.23763908} } machinelearning datascience ai deeplearning python