# Deletion Tuning: Matching Full Fine-Tuning by Zeroing Under 0.05% of Weights

> Source: <https://explainx.ai/blog/deletion-tuning-llm-post-training-neurips-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-11 00:00:00+00:00

**Deletion Tuning asks a blunt question: how little of a pretrained LLM must change to learn a new skill?** The NeurIPS 2026 paper [arXiv:2611.10866](https://arxiv.org/abs/2611.10866?ref=explainx) by **Jack Lu**, **Zhenbang Yang**, **Mike Lasby**, and colleagues answers with a surprising magnitude: **under 0.05%** of weights—**zeroed, not updated**—can **match full fine-tuning (FFT)** across **RL with verifiable rewards**, **supervised fine-tuning**, and **on-policy distillation**.

Hacker News surfaced the result on November 10 because it challenges the default playbook: **dense FFT** for quality, **LoRA** for efficiency. Deletion Tuning offers a third knob: **extreme sparsity via learned deletion masks** on **MLP gate projections**.

## TL;DR

| Topic | Finding | 
|---|---|
| Method | Learned **binary mask** ; weights set to**zero** | 
| Sparsity | **Under 0.05%** of parameters touched | 
| Quality | Matches **FFT** in three post-training regimes | 
| Checkpoint | Mask **hundreds–thousands× smaller** than FFT | 
| Forgetting | **Lower** than FFT, LoRA, random sparse FT in RL tests | 
| Analysis | **Stable rank** , spectrum drift, principal-weight overlap vs baselines | 

## Mechanism: delete instead of delta

Modern transformers route MLP blocks through **gated feed-forward** paths. Deletion Tuning trains a **mask** that selects which gate weights become **exactly zero** after post-training. No low-rank adapters, no full-rank gradient on the entire matrix—just **discrete structure**.

Two practical consequences follow immediately:

1. **Storage:** ship a bitmask instead of a multi-gigabyte FFT delta.
2. **Continual learning:** prior capabilities see less interference because fewer active weights move.

The paper’s RL experiments report **prior-task accuracy** near the **base model**, while FFT, LoRA, and **random sparse fine-tuning** dropped up to **5.4 points** on earlier tasks. For teams stacking **monthly skill adapters**, that forgetting gap matters more than a leaderboard point.

Fine-tuning adapter collar on a model core illustrating sparse Deletion Tuning weight masks

## Where it matches FFT

Authors evaluate at scales where FFT remains the **gold standard**:

| Regime | Why labs still use FFT | 
|---|---|
| **RLVR** | Reward noise needs capacity | 
| **SFT** | Imitation of curated dialogues | 
| **On-policy distillation** | Teacher-student alignment with rollout coupling | 

Matching FFT with **0.05% sparsity** is not a toy MNIST result—it claims parity on **LLM post-training** stacks teams actually run after pretraining.

## Geometry analysis: why masking can approximate dense updates

The paper compares **Deletion Tuning**, **LoRA**, and **random sparse fine-tuning** against FFT using:

- **Stable rank** of weight changes
- **Spectrum drift**
- **Principal-weight overlap**

Deletion Tuning lands **closest to FFT** on these metrics in the reported RL setting. Intuition for practitioners: LoRA constrains updates to a **low-rank subspace**, which helps efficiency but can miss FFT’s **direction** in weight space. Learned deletion may recover **FFT-aligned** subspaces while touching **few coordinates**.

## Comparison table for engineering leads

| Approach | Train cost | Artifact size | Forgetting risk | Maturity | 
|---|---|---|---|---|
| FFT | High | Huge | Higher | Production default | 
| LoRA | Medium | Small | Medium | Ubiquitous | 
| QLoRA | Medium-low | Small | Medium | Ubiquitous | 
| Deletion Tuning | Mask search | **Tiny mask** | **Lower in paper** | Research-stage | 

Do not rip out LoRA pipelines overnight. Deletion Tuning adds **mask learning** overhead and needs **replication on your reward model** before production claims.

## How to try it responsibly

1. **Reproduce** on a public base model with the authors’ hyperparameters once code drops.
2. Benchmark against **your** LoRA rank and FFT baselines on the**same eval harness** —[Bedrock eval playbooks](https://explainx.ai/blog/amazon-bedrock-agentcore-evaluation-playbook-deepeval-autoeval-guide-2026) are a template.
3. Track **prior-task regression** when stacking two skills; that is where the paper claims advantage.
4. Export **only masks** to edge devices if storage is the bottleneck—not the full teacher delta.

## Links to explainx.ai training discourse

- [Microsoft Lazarus dormant neuron backdoor editing](https://explainx.ai/blog/microsoft-lazarus-dormant-neuron-agent-backdoor-weight-editing-2026) — different weight edit philosophy, security not efficiency
- [FP4 pretraining rare facts](https://explainx.ai/blog/fp4-pretraining-rare-facts-nvfp4-loss-curve-2026) — precision trade-offs in training stacks
- [Anthropic homeopathic learning poisoned fine-tuning](https://explainx.ai/blog/anthropic-homeopathic-learning-poisoned-fine-tuning-2026) — why post-training data hygiene still dominates
- [Remote Labor Index automation](https://explainx.ai/blog/remote-labor-index-fable-5-2-automation-rate-scale-cais-2026) — economic frame for cheaper adaptation
- [AI labs output token inflation](https://explainx.ai/blog/ai-labs-output-token-inflation-pricing-playcode-2026) — serving economics adjacent to adapter size

## RLVR, SFT, and distillation: what “matches FFT” implies

**Reinforcement learning with verifiable rewards** is the regime where labs pour GPU months after base pretraining. Deletion Tuning’s claim is not that masking beats a bigger base model—it is that masking reaches **the same post-training lift** as updating every weight. For coding rewards, that means equal pass rates on verifiable unit tests at equal sample budgets in the paper’s setup.

**Supervised fine-tuning** still dominates product teams shipping style and safety layers. A mask that matches FFT while storing **kilobytes** instead of **gigabytes** matters for **multi-tenant SaaS** that keeps one base weights blob and many customer masks—if serving stacks can apply masks efficiently.

**On-policy distillation** couples teacher rollouts to student updates. Sparse deletion may reduce **student drift** when the teacher changes slightly between curriculum weeks—another angle on the forgetting table.

## Serving and MLOps questions not answered in v1

The paper optimizes **training science**, not **inference kernels**. Before production:

- Can your stack **apply binary masks** on gate projections at runtime without unfused slowdown?
- Does the mask require **re-merging** into base weights for TensorRT or vLLM, defeating storage wins?
- How do masks compose when **two skills** must be active—union, intersection, or sequential load?

Until serving vendors document mask ops, treat Deletion Tuning as **research** for teams that own custom CUDA or **internal inference**.

## Experiment recipe once code is public

1. Pick a **7B–13B** base with public RLVR or SFT scripts you already trust.
2. Train **FFT** ,**LoRA rank 64** , and**Deletion Tuning** with matched steps and data.
3. Log **prior-task eval** before and after each skill add.
4. Store **artifact bytes** and**load time** to prod inference mock.
5. Publish negative results—masks may fail on small models with narrow MLP gates.

## Historical context: sparsity waves in NLP

Pruning and **Lottery Ticket** narratives predate LLMs. Deletion Tuning differs by learning **which gates to zero during post-training** rather than pruning a finished model for inference only. That connects to [Microsoft Lazarus](https://explainx.ai/blog/microsoft-lazarus-dormant-neuron-agent-backdoor-weight-editing-2026) weight editing—but for **efficiency** instead of backdoors.

## Limitations

- **0.05%** is an order-of-magnitude headline; your task may need denser masks.
- **Gate-only** edits may not cover attention layers your task requires.
- **Binary masks** can be harder to serve in fused kernels without custom CUDA.
- NeurIPS version may differ from arXiv v1—watch camera-ready for code.

## What Hacker News readers will debate

The November 10 HN thread had **no substantive comments** yet—common for fresh arXiv drops—but the title alone invites predictable arguments:

- **“0.05% sounds like lottery tickets”** — authors counter with**geometry metrics** aligned to FFT, not random masks.
- **“LoRA already solved this”** — forgetting table is the rebuttal for**multi-task RL** .
- **“Can HF run this?”** — serving integration unknown until code ships.

When commenting yourself, distinguish **training-time deletion** from **inference pruning** products.

## Glossary for cross-team meetings

| Term | Plain meaning | 
|---|---|
| **Gate projection** | MLP switch layer in transformer blocks | 
| **Deletion mask** | Binary vector forcing weights to zero | 
| **RLVR** | RL with programmatic verifiers (unit tests, compilers) | 
| **Stable rank** | Measure of how “wide” weight updates are | 
| **Catastrophic forgetting** | New skill destroys old skill metrics | 

## Paper details and authors

- **Venue:** NeurIPS 2026 (per arXiv comments)
- **Affiliations:** New York University, University of Calgary, and collaborators per arXiv
- **DOI:**[10.48550/arXiv.2611.10866](https://doi.org/10.48550/arXiv.2611.10866?ref=explainx)
- **v1 date:** November 7, 2026

Watch author GitHub for **mask export format** and **license**.

## Decision tree: FFT vs LoRA vs Deletion Tuning

| If your priority is… | Start with… | 
|---|---|
| Maximum single-task quality | FFT baseline | 
| Fast iteration, mature tools | LoRA / QLoRA | 
| Tiny artifacts + low forgetting | Pilot Deletion Tuning when code drops | 
| Regulatory audit of deltas | Deletion masks may log cleaner than rank-64 adapters | 

Re-evaluate after **two skills** land; forgetting shows up late.

## Connections to lab economics

[Anthropic gross margin filings](https://explainx.ai/blog/anthropic-s-1-gross-margin-85-percent-serving-compute-vs-all-compute-2026) show post-training and serving dominate spend. Methods that shrink **adapter storage** and **reload time** matter at millions of tenants—even before they change headline benchmark scores.

Teams running **weekly RLHF** on the same base should A/B **mask count** against **LoRA rank** on a fixed verifier suite before touching production traffic. Document negative outcomes; the paper’s positive regime may not transfer to your reward hacking-prone environment.

For **on-device** finetunes, deletion masks may be attractive if merged weights exceed mobile RAM— but mobile stacks rarely expose arbitrary gate edits today.

Read the PDF **appendix figures** before citing 0.05% in executive slides; sparsity varies by layer depth and training phase. Pair qualitative samples with **[Opus formalization work](https://explainx.ai/blog/opus-5-5-cpython-lean-formalization-418-bugs-2026)** only as a reminder that small mechanical changes can surface large behavioral shifts—you still need task evals, not Mathlib analogies alone.

Security reviewers may ask whether sparse masks introduce **new attack surface** for weight poisoning. Treat masks like any adapter artifact: sign them, version them, and scan checkpoints in CI—the same discipline we describe for [fragile agent-edited codebases](https://explainx.ai/blog/ai-usage-fragile-codebase-agentic-coding-hn-2026). Until serving kernels ship first-class mask ops, deletion checkpoints remain a training-room technique, not a fleet-wide rollout. Revisit this page when authors post inference benchmark numbers.

## Bottom line for practitioners

The practical takeaway is narrow but useful. Deletion Tuning shows that post-training gains do not always need dense updates. A tiny, learned set of zeroed gate weights can reach the same quality as full fine-tuning in the paper's setups, and it keeps earlier skills more intact. Treat it as a strong research signal, not a drop-in replacement. Replicate it on your own data, check serving support for masks, and compare it with your current LoRA baseline before you change any production pipeline.

## Related reading

*Paper metadata reflects arXiv 2611.10866 v1 submitted November 7, 2026.*