# Zero Forgetting in LLMs? We Benchmarked Null-Space Projection on SmolLM and Hit the Plasticity Wall

> Source: <https://dev.to/akshatraj00/zero-forgetting-in-llms-we-benchmarked-null-space-projection-on-smollm-and-hit-the-plasticity-wall-4n7a>
> Published: 2026-10-29 12:06:32+00:00

Why orthogonal projections eliminate catastrophic forgetting in autoregressive Transformers—and the mathematical reason behind the 62% downstream plasticity penalty.

Fine-tuning Large Language Models sequentially without retraining from scratch is notoriously broken. When you adapt an LLM to **Task B** after **Task A**, it systematically overwrites the weight configurations that enabled performance on Task A. This is **Catastrophic Forgetting**.

While Parameter-Efficient Fine-Tuning (PEFT) techniques like **LoRA** (Low-Rank Adaptation) isolate task-specific updates into small low-rank matrices:

$$\Delta W = \frac{\alpha}{r} B A$$

a critical bottleneck arises during deployment: **How do you consolidate multiple sequential adapters into a single base model without incurring catastrophic interference?**

Directly merging delta weights:

$$W_{\text{base}} \leftarrow W_{\text{base}} + \Delta W$$

destroys historical task performance.

To evaluate this frontier, we implemented and benchmarked **Wake-Sleep Null-Space Consolidation (WS-NSC)** on **SmolLM-135M**. The code is open-sourced, and the complete benchmark data is permanently registered under DOI **[10.5281/zenodo.23763908](https://doi.org/10.5281/zenodo.23763908)**.

Here is the empirical reality of what happens when you mathematically force parameter updates into the orthogonal null-space of prior activations.

The framework operates in a biologically inspired two-phase regime:

```
[Wake Phase: Online LoRA Training]
            │
            ▼
[Activation Manifold Collection: C = Σ (x^T x) / M]
            │
            ▼
[Sleep Phase: Null-Space Orthogonal Projection]
            │
            ▼
[Consolidation: W_base ← W_base + ΔW @ P_null]
```

The model freezes base parameters $W_0$ and trains a standard LoRA adapter $(A_t, B_t)$ on the active task stream $\mathcal{D}_t$.

Forward hooks register linear input activations $x \in \mathbb{R}^{B \times L \times d_{\text{in}}}$ across reference tokens and historical tasks. We compute the uncentered empirical second-moment covariance matrix:

$$C = \frac{1}{M} \sum_{i=1}^M x_i^\top x_i \in \mathbb{R}^{d_{\text{in}} \times d_{\text{in}}}$$

We extract eigenvectors $U$ and eigenvalues $\Lambda$ such that $C = U \Lambda U^\top$. We retain the top-$k$ principal components accounting for a variance threshold $\rho \in [0.90, 0.95]$:

$$\frac{\sum_{i=1}^k \lambda_i}{\sum_{j=1}^d \lambda_j} \ge \rho$$

The parameter update $\Delta W$ is then projected onto the exact null-space of $U_k$:

$$P_{\text{null}} = I - U_k U_k^\top$$

$$\Delta W_{\text{safe}} = \Delta W \cdot P_{\text{null}}$$

Because $x U_k \approx x$ for historical activations, projecting into $P_{\text{null}}$ ensures:

$$\Delta W_{\text{safe}} \cdot x^\top \approx 0$$

Theoretical guarantee: **Prior task forward passes experience zero functional shift.**

We configured a 4-stage sequential continual learning setup:

All experiments were executed on an A100 GPU using PyTorch AMP across all 7 projection matrices (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`).

| Consolidation Strategy | Wake Gain (%) | Post-Sleep Gain (%) | Plasticity Retention | Forgetting (nats) | Relative Forgetting (%) | Anchor Drift $\Delta$ | Protected Rank ($k/d$) | Update Norm Kept | 
|---|---|---|---|---|---|---|---|---|
| **Direct-Merge** | **16.35%** | **16.32%** | **99.8%** | $+0.103$ | $+5.833\%$ | $+0.057$ | $0.000$ | $100.0\%$ | 
| **WS-NSC ($\rho=0.90$)** | 15.29% | 5.76% | **37.7%** | $\mathbf{-0.007}$ | $\mathbf{-0.390\%}$ | $\mathbf{-0.005}$ | $36.2\%$ | $73.9\%$ | 
| **WS-NSC ($\rho=0.95$)** | 15.33% | 3.74% | **24.4%** | $\mathbf{-0.007}$ | $-0.387\%$ | $-0.002$ | $54.5\%$ | $61.6\%$ | 

In direct merging, catastrophic forgetting is glaring: backward tasks degrade by $+5.83\%$, and the general-domain anchor drifts by $+0.057$ nats.

Under WS-NSC ($\rho=0.90$), backward forgetting drops to **$-0.39\%$** and anchor drift drops to **$-0.005$ nats**. Historical knowledge is preserved without storing or replaying raw pretraining datasets.

While stability is solved, it incurs an acute functional cost. Under $\rho=0.90$, **$36.2\%$ of the linear subspace is locked down**, causing post-sleep task performance to retain only **$37.7\%$** of what was acquired during the wake phase.

Pushing protection to $\rho=0.95$ locks **$54.5\%$** of the rank, collapsing retained gains down to **$24.4\%$**. In deep nonlinear architectures, hard orthogonal projection chokes forward model capacity.

A common assumption in model merging is that preserving weight magnitude preserves task utility.

Our data disproves this: **retaining $73.9\%$ of parameter update norm resulted in retaining only $37.7\%$ of task gain.** Spectral truncation clips the highest-energy activation vectors needed for downstream task execution.

The full codebase, automated plotting pipelines, and raw execution logs are open for external verification:

To reproduce the benchmark locally:

```
git clone https://github.com/AkshatRaj00/ws-nsc.git
cd ws-nsc
pip install -r requirements.txt

# Run empirical sequential evaluation
python src/ws_nsc.py --model HuggingFaceTB/SmolLM-135M --amp --batch-size 4 --eval-batch-size 4 --var-retention 0.90 0.95

# Generate loss trajectories & Pareto frontier plots
python src/plot_results.py
@article{raj2026wsnsc,
  title={Wake-Sleep Null-Space Consolidation: Quantifying the Stability-Plasticity Frontier in Sequential LLM Adaptation},
  author={Raj, Akshat},
  year={2026},
  publisher={Zenodo},
  doi={10.5281/zenodo.23763908},
  url={https://doi.org/10.5281/zenodo.23763908}
}
```

`machinelearning` `datascience` `ai` `deeplearning` `python`