cd /news/large-language-models/this-weight-does-not-spark-joy · home › topics › large-language-models › article
[ARTICLE · art-172184] src=ianbarber.blog ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

This weight does not spark joy

A paper by Lu et al., "The Surprising Effectiveness of Deleting Weights in LLM Post-Training," finds that zeroing out gate weights in gated linear units can fine-tune large language models using masks under 0.05% of model weights, producing adapters orders of magnitude smaller than full fine-tuning checkpoints while retaining prior-task accuracy. In an independent replication fine-tuning Liquid's LFM3-1.7B on Wordle, the author measured 81% accuracy with deletion tuning versus 69% for full fine-tuning and 68% for LoRA, though across only a few seeds, and found all three methods stayed within one point of the base model on the paper's benchmark suite at recommended learning rates. The result matters because it suggests a far cheaper, less destructive alternative to full fine-tuning and LoRA for adjusting model behavior.

recommendation

For behavior fine-tuning, try masking gate-matrix weights instead of full fine-tuning or LoRA: the source's deletion masks were under 0.05% of weights and stayed within one point of the base model at recommended learning rates.

more in playbook: Fine-tuning and data
by read4 min views7 publishedNov 18, 2026
This weight does not spark joy
Image: Ianbarber (auto-discovered)

The bulk of parameters in LLMs are in the feed-forward networks at the end of the transformer block. Because these are linear layers, they need to be separated by an activation function to avoid collapsing into a single transform.

In the old days of ML that was usually a sigmoid, so you’d have W2 · sigmoid(W1 · x), where x was the input. Then it became clear that we could get away with something a lot simpler and ReLU became standard. ReLU is just max(0, value), but you can think of it as a gate: W1 is multiplied with the input, we zero any values less than zero, and the result is multiplied with W2.

The problem with that is that the W1 matrix is both setting the gate value, and the value-value. Gated Linear Units add an extra matrix: W2 · ((W1 · x) * sigmoid(Gate · x)). That lets the model learn with one one set of parameters what input<sup>1</sup> should be let through, and with another how that input should be transformed.

Fine-tuning a model involves modifying weights to adjust behaviors, so it is a little surprising to find that you can restrict yourself to just zeroing some of those gate weights, and still effectively fine-tune a model! “The Surprising Effectiveness of Deleting Weights in LLM Post-Training”, from Lu et al.:

Deletion Tuning is compact and forgets less. Its masks are orders of magnitude smaller than FFT checkpoints in all three regimes, and it keeps prior-task accuracy throughout training.

“FFT” in the quote above refers to full fine-tuning, where every parameter can be updated. They also compared LoRA, where you learn an adapter that is added to the weight matrices. They found their method<sup>2</sup> ended up both with a smaller adapter (a bit-mask for the gate matrices), and impacting the base model’s capabilities less than the traditional fine-tuning approaches.

This is odd! My read is that it is effectively suppressing “wrong” behaviors, but in a way that doesn’t affect the overall shape of the learned manifold too much, hence the lowered impact. That said, deletion doesn’t mean we are just turning things off: some weights are negative, so deleting them can open a gate up rather than close it.

It turns out, there is quite a bit of precedent for this (as the paper properly cites!) Masking out the weights is a form of weight adjustment, and there is some prior work in vision models and later in BERT-style language models, that suggested masking can match full fine-tuning. The authors here call out how sparse their mask is though, under 0.05% of the model’s weights.

I tried this by fine-tuning Liquid’s excellent LFM3-1.7B to learn Wordle<sup>3</sup>, and yeah, it does seem to work as well as full fine-tuning! The base model got nothing right, deletion tuning got that to 81% versus 69% for FFT and 68% for LoRA, though with just a few seeds so I wouldn’t put too much stock in those precise numbers.

With regards to degradation on the base model, at the learning rates the paper recommends everything was fine: all three stayed within a point of the base model on the paper’s benchmark suite. Cranking up the learning rate did turn the model into a wordle-only machine under full-fine tuning, and only slightly less so with deletion. LoRA, if anything, held up best.

I also didn’t quite get the sparsity they saw with their RL setup: the mask was around 4MB, which is similar to a rank 1 LoRA<sup>4</sup>. I did end up with much less sparse mask though. The harder you push the learning rate the more deletions I got, so I’d imagine with some tuning it could be somewhat sparser.

What was surprising was how arbitrary the mask seemed to be though. Two runs with different seeds got to the same score but only shared about 6% of their deletions. They were impacting the same blocks, but not the same specific weights. It seems to matter where you thin, but not so much which exact weights get Marie Kondo’d.

  1. What, but also “how much”. Sigmoid is 0-1, but everyone these days uses a Swish activation with their Gated Linear Unit, SwiGLU. Swish is unbounded at the top so the gate branch also scales the value. ↩︎
  2. Under RL: my experiment a bit later is SFT, so its not apples to apples. ↩︎
  3. Or, as it turned out, learn Wordle for the specific words in the SFT Wordle training set, which is… not much use. But then what use is a model that plays Wordle? I was so preoccupied with whether or not I could, I didn’t stop to think if I should. ↩︎
  4. I think you can probably get something akin to full fine-tuning performance for the storage cost of a LoRA, which is pretty great, but might not buy you that much .↩︎
── more in #large-language-models 4 stories · sorted by recency
── more on @lu et al. 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/this-weight-does-not…] indexed:0 read:4min 2026-11-18 · —