{"slug": "deletion-tuning-matching-full-fine-tuning-by-zeroing-under-0-05-of-weights", "title": "Deletion Tuning: Matching Full Fine-Tuning by Zeroing Under 0.05% of Weights", "summary": "A NeurIPS 2026 paper, arXiv:2611.10866, by Jack Lu, Zhenbang Yang, Mike Lasby and colleagues reports that zeroing under 0.05% of a pretrained LLM's weights — rather than updating them — can match full fine-tuning (FFT) across reinforcement learning with verifiable rewards, supervised fine-tuning and on-policy distillation. The method, Deletion Tuning, learns a binary mask that sets selected MLP gate projection weights exactly to zero, producing checkpoints hundreds to thousands of times smaller than an FFT delta. In the paper's RL experiments, prior-task accuracy stayed near the base model while FFT, LoRA and random sparse fine-tuning dropped up to 5.4 points on earlier tasks.", "body_md": "**Deletion Tuning asks a blunt question: how little of a pretrained LLM must change to learn a new skill?** The NeurIPS 2026 paper [arXiv:2611.10866](https://arxiv.org/abs/2611.10866?ref=explainx) by **Jack Lu**, **Zhenbang Yang**, **Mike Lasby**, and colleagues answers with a surprising magnitude: **under 0.05%** of weights—**zeroed, not updated**—can **match full fine-tuning (FFT)** across **RL with verifiable rewards**, **supervised fine-tuning**, and **on-policy distillation**.\n\nHacker News surfaced the result on November 10 because it challenges the default playbook: **dense FFT** for quality, **LoRA** for efficiency. Deletion Tuning offers a third knob: **extreme sparsity via learned deletion masks** on **MLP gate projections**.\n\n## TL;DR\n\n| Topic | Finding | \n|---|---|\n| Method | Learned **binary mask** ; weights set to**zero** | \n| Sparsity | **Under 0.05%** of parameters touched | \n| Quality | Matches **FFT** in three post-training regimes | \n| Checkpoint | Mask **hundreds–thousands× smaller** than FFT | \n| Forgetting | **Lower** than FFT, LoRA, random sparse FT in RL tests | \n| Analysis | **Stable rank** , spectrum drift, principal-weight overlap vs baselines | \n\n## Mechanism: delete instead of delta\n\nModern transformers route MLP blocks through **gated feed-forward** paths. Deletion Tuning trains a **mask** that selects which gate weights become **exactly zero** after post-training. No low-rank adapters, no full-rank gradient on the entire matrix—just **discrete structure**.\n\nTwo practical consequences follow immediately:\n\n1. **Storage:** ship a bitmask instead of a multi-gigabyte FFT delta.\n2. **Continual learning:** prior capabilities see less interference because fewer active weights move.\n\nThe paper’s RL experiments report **prior-task accuracy** near the **base model**, while FFT, LoRA, and **random sparse fine-tuning** dropped up to **5.4 points** on earlier tasks. For teams stacking **monthly skill adapters**, that forgetting gap matters more than a leaderboard point.\n\nFine-tuning adapter collar on a model core illustrating sparse Deletion Tuning weight masks\n\n## Where it matches FFT\n\nAuthors evaluate at scales where FFT remains the **gold standard**:\n\n| Regime | Why labs still use FFT | \n|---|---|\n| **RLVR** | Reward noise needs capacity | \n| **SFT** | Imitation of curated dialogues | \n| **On-policy distillation** | Teacher-student alignment with rollout coupling | \n\nMatching FFT with **0.05% sparsity** is not a toy MNIST result—it claims parity on **LLM post-training** stacks teams actually run after pretraining.\n\n## Geometry analysis: why masking can approximate dense updates\n\nThe paper compares **Deletion Tuning**, **LoRA**, and **random sparse fine-tuning** against FFT using:\n\n- **Stable rank** of weight changes\n- **Spectrum drift**\n- **Principal-weight overlap**\n\nDeletion Tuning lands **closest to FFT** on these metrics in the reported RL setting. Intuition for practitioners: LoRA constrains updates to a **low-rank subspace**, which helps efficiency but can miss FFT’s **direction** in weight space. Learned deletion may recover **FFT-aligned** subspaces while touching **few coordinates**.\n\n## Comparison table for engineering leads\n\n| Approach | Train cost | Artifact size | Forgetting risk | Maturity | \n|---|---|---|---|---|\n| FFT | High | Huge | Higher | Production default | \n| LoRA | Medium | Small | Medium | Ubiquitous | \n| QLoRA | Medium-low | Small | Medium | Ubiquitous | \n| Deletion Tuning | Mask search | **Tiny mask** | **Lower in paper** | Research-stage | \n\nDo not rip out LoRA pipelines overnight. Deletion Tuning adds **mask learning** overhead and needs **replication on your reward model** before production claims.\n\n## How to try it responsibly\n\n1. **Reproduce** on a public base model with the authors’ hyperparameters once code drops.\n2. Benchmark against **your** LoRA rank and FFT baselines on the**same eval harness** —[Bedrock eval playbooks](https://explainx.ai/blog/amazon-bedrock-agentcore-evaluation-playbook-deepeval-autoeval-guide-2026) are a template.\n3. Track **prior-task regression** when stacking two skills; that is where the paper claims advantage.\n4. Export **only masks** to edge devices if storage is the bottleneck—not the full teacher delta.\n\n## Links to explainx.ai training discourse\n\n- [Microsoft Lazarus dormant neuron backdoor editing](https://explainx.ai/blog/microsoft-lazarus-dormant-neuron-agent-backdoor-weight-editing-2026) — different weight edit philosophy, security not efficiency\n- [FP4 pretraining rare facts](https://explainx.ai/blog/fp4-pretraining-rare-facts-nvfp4-loss-curve-2026) — precision trade-offs in training stacks\n- [Anthropic homeopathic learning poisoned fine-tuning](https://explainx.ai/blog/anthropic-homeopathic-learning-poisoned-fine-tuning-2026) — why post-training data hygiene still dominates\n- [Remote Labor Index automation](https://explainx.ai/blog/remote-labor-index-fable-5-2-automation-rate-scale-cais-2026) — economic frame for cheaper adaptation\n- [AI labs output token inflation](https://explainx.ai/blog/ai-labs-output-token-inflation-pricing-playcode-2026) — serving economics adjacent to adapter size\n\n## RLVR, SFT, and distillation: what “matches FFT” implies\n\n**Reinforcement learning with verifiable rewards** is the regime where labs pour GPU months after base pretraining. Deletion Tuning’s claim is not that masking beats a bigger base model—it is that masking reaches **the same post-training lift** as updating every weight. For coding rewards, that means equal pass rates on verifiable unit tests at equal sample budgets in the paper’s setup.\n\n**Supervised fine-tuning** still dominates product teams shipping style and safety layers. A mask that matches FFT while storing **kilobytes** instead of **gigabytes** matters for **multi-tenant SaaS** that keeps one base weights blob and many customer masks—if serving stacks can apply masks efficiently.\n\n**On-policy distillation** couples teacher rollouts to student updates. Sparse deletion may reduce **student drift** when the teacher changes slightly between curriculum weeks—another angle on the forgetting table.\n\n## Serving and MLOps questions not answered in v1\n\nThe paper optimizes **training science**, not **inference kernels**. Before production:\n\n- Can your stack **apply binary masks** on gate projections at runtime without unfused slowdown?\n- Does the mask require **re-merging** into base weights for TensorRT or vLLM, defeating storage wins?\n- How do masks compose when **two skills** must be active—union, intersection, or sequential load?\n\nUntil serving vendors document mask ops, treat Deletion Tuning as **research** for teams that own custom CUDA or **internal inference**.\n\n## Experiment recipe once code is public\n\n1. Pick a **7B–13B** base with public RLVR or SFT scripts you already trust.\n2. Train **FFT** ,**LoRA rank 64** , and**Deletion Tuning** with matched steps and data.\n3. Log **prior-task eval** before and after each skill add.\n4. Store **artifact bytes** and**load time** to prod inference mock.\n5. Publish negative results—masks may fail on small models with narrow MLP gates.\n\n## Historical context: sparsity waves in NLP\n\nPruning and **Lottery Ticket** narratives predate LLMs. Deletion Tuning differs by learning **which gates to zero during post-training** rather than pruning a finished model for inference only. That connects to [Microsoft Lazarus](https://explainx.ai/blog/microsoft-lazarus-dormant-neuron-agent-backdoor-weight-editing-2026) weight editing—but for **efficiency** instead of backdoors.\n\n## Limitations\n\n- **0.05%** is an order-of-magnitude headline; your task may need denser masks.\n- **Gate-only** edits may not cover attention layers your task requires.\n- **Binary masks** can be harder to serve in fused kernels without custom CUDA.\n- NeurIPS version may differ from arXiv v1—watch camera-ready for code.\n\n## What Hacker News readers will debate\n\nThe November 10 HN thread had **no substantive comments** yet—common for fresh arXiv drops—but the title alone invites predictable arguments:\n\n- **“0.05% sounds like lottery tickets”** — authors counter with**geometry metrics** aligned to FFT, not random masks.\n- **“LoRA already solved this”** — forgetting table is the rebuttal for**multi-task RL** .\n- **“Can HF run this?”** — serving integration unknown until code ships.\n\nWhen commenting yourself, distinguish **training-time deletion** from **inference pruning** products.\n\n## Glossary for cross-team meetings\n\n| Term | Plain meaning | \n|---|---|\n| **Gate projection** | MLP switch layer in transformer blocks | \n| **Deletion mask** | Binary vector forcing weights to zero | \n| **RLVR** | RL with programmatic verifiers (unit tests, compilers) | \n| **Stable rank** | Measure of how “wide” weight updates are | \n| **Catastrophic forgetting** | New skill destroys old skill metrics | \n\n## Paper details and authors\n\n- **Venue:** NeurIPS 2026 (per arXiv comments)\n- **Affiliations:** New York University, University of Calgary, and collaborators per arXiv\n- **DOI:**[10.48550/arXiv.2611.10866](https://doi.org/10.48550/arXiv.2611.10866?ref=explainx)\n- **v1 date:** November 7, 2026\n\nWatch author GitHub for **mask export format** and **license**.\n\n## Decision tree: FFT vs LoRA vs Deletion Tuning\n\n| If your priority is… | Start with… | \n|---|---|\n| Maximum single-task quality | FFT baseline | \n| Fast iteration, mature tools | LoRA / QLoRA | \n| Tiny artifacts + low forgetting | Pilot Deletion Tuning when code drops | \n| Regulatory audit of deltas | Deletion masks may log cleaner than rank-64 adapters | \n\nRe-evaluate after **two skills** land; forgetting shows up late.\n\n## Connections to lab economics\n\n[Anthropic gross margin filings](https://explainx.ai/blog/anthropic-s-1-gross-margin-85-percent-serving-compute-vs-all-compute-2026) show post-training and serving dominate spend. Methods that shrink **adapter storage** and **reload time** matter at millions of tenants—even before they change headline benchmark scores.\n\nTeams running **weekly RLHF** on the same base should A/B **mask count** against **LoRA rank** on a fixed verifier suite before touching production traffic. Document negative outcomes; the paper’s positive regime may not transfer to your reward hacking-prone environment.\n\nFor **on-device** finetunes, deletion masks may be attractive if merged weights exceed mobile RAM— but mobile stacks rarely expose arbitrary gate edits today.\n\nRead the PDF **appendix figures** before citing 0.05% in executive slides; sparsity varies by layer depth and training phase. Pair qualitative samples with **[Opus formalization work](https://explainx.ai/blog/opus-5-5-cpython-lean-formalization-418-bugs-2026)** only as a reminder that small mechanical changes can surface large behavioral shifts—you still need task evals, not Mathlib analogies alone.\n\nSecurity reviewers may ask whether sparse masks introduce **new attack surface** for weight poisoning. Treat masks like any adapter artifact: sign them, version them, and scan checkpoints in CI—the same discipline we describe for [fragile agent-edited codebases](https://explainx.ai/blog/ai-usage-fragile-codebase-agentic-coding-hn-2026). Until serving kernels ship first-class mask ops, deletion checkpoints remain a training-room technique, not a fleet-wide rollout. Revisit this page when authors post inference benchmark numbers.\n\n## Bottom line for practitioners\n\nThe practical takeaway is narrow but useful. Deletion Tuning shows that post-training gains do not always need dense updates. A tiny, learned set of zeroed gate weights can reach the same quality as full fine-tuning in the paper's setups, and it keeps earlier skills more intact. Treat it as a strong research signal, not a drop-in replacement. Replicate it on your own data, check serving support for masks, and compare it with your current LoRA baseline before you change any production pipeline.\n\n## Related reading\n\n*Paper metadata reflects arXiv 2611.10866 v1 submitted November 7, 2026.*", "url": "https://wpnews.pro/news/deletion-tuning-matching-full-fine-tuning-by-zeroing-under-0-05-of-weights", "canonical_source": "https://explainx.ai/blog/deletion-tuning-llm-post-training-neurips-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-11 00:00:00+00:00", "updated_at": "2026-11-11 04:22:05.261295+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-tools"], "entities": ["Jack Lu", "Zhenbang Yang", "Mike Lasby", "NeurIPS 2026", "arXiv:2611.10866", "Deletion Tuning", "LoRA", "Hacker News"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deletion-tuning-matching-full-fine-tuning-by-zeroing-under-0-05-of-weights", "markdown": "https://wpnews.pro/news/deletion-tuning-matching-full-fine-tuning-by-zeroing-under-0-05-of-weights.md", "text": "https://wpnews.pro/news/deletion-tuning-matching-full-fine-tuning-by-zeroing-under-0-05-of-weights.txt", "jsonld": "https://wpnews.pro/news/deletion-tuning-matching-full-fine-tuning-by-zeroing-under-0-05-of-weights.jsonld"}}