If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent changes to weights if the change is at least ~1/512 of the weight itself, so at small learning rates only the smallest weights can update.
As a result, you have a sparse fine-tuner that trains the small-magnitude weights and freezes the rest, with the cutoff set by your learning rate. This is a thing - I found half a dozen peer-reviewed papers (with thousands of citations) that fine-tune this way, but none of them reference this behaviour.
I dug into one called Quiet-STaR, because (a) it's a cool paper, (b) they released their weights, and (c) their code works! So I diffed their base model and saw that indeed the small-magnitude weights were the only ones updated during their training. Then I reran its recipe with an optimizer that applies every update, and watched it diverge. A higher-precision optimizer needs a 3 to 10 times smaller learning rate to get the same result, and I couldn't achieve a better result.
This started with Barron, Gnanapragasam & Tschannen's "A Fix for Softmax and Cross-Entropy" (Oct 2026). They show that the standard way of computing cross-entropy rounds the correct answer's gradient to exactly zero once the model is confident enough (past a margin of about 5.5 in bf16, which they call the precision cliff), and that this, interacting with Adam, is what produces the "slingshot" loss spikes seen during grokking. Fix the rounding and the spikes go away: something people had studied as training dynamics was rounding. That made me wonder what else out there is weird like this. This is one, and it runs the other way: their rounding bug adds instability to training; this one hides it.
The code, logs and evaluation results are in mweiss17/quietstar-bf16-adamw.
1. How bf16 affects optimization #
A floating-point number has two parts that matter here. The exponent picks a power-of-two interval (1 to 2, 2 to 4, 4 to 8, and so on), and the mantissa picks one of a fixed number of evenly spaced slots inside it. Every interval gets the same number of slots, however wide it is. So the values sit close together near zero and spread out as numbers get bigger: the gap between neighbouring values doubles every time you cross a power of two. Figure 1 shows this with a toy format that has only 4 slots per interval.
bf16 and fp32 both have 8 exponent bits, so they cover the same range of numbers. The difference is the mantissa. fp32 has 23 mantissa bits, so 8,388,608 slots per interval. bf16 keeps only the top 7, so 128. A bf16 number is the first half of the fp32 number, rounded. Its gaps are 65,536 times wider: about 1/128 to 1/256 of the number itself.
Now add a small change to a weight, as an optimizer does on every step. The result has to be rounded to a value the format can hold. If the change is less than half a gap, it rounds back to where it started and the update is lost. Adam's step is roughly the learning rate, so at 1e-6 a typical Mistral weight loses its update while a weight eight times smaller keeps it.
To see what this does to training, go back to the toy format from Figure 1 and run Adam on two weights, w₁ and w₂. Every dot is a value the format can hold. Two copies of Adam start from the same point and head for the same minimum. One (orange) keeps its weights in the toy format and rounds after every step. The other (blue) keeps a full-precision copy.
bf16 is the same picture with 128 values per interval instead of 4, so a weight loses any change smaller than roughly 1/512 of its own size.
2. How bf16 messes with AdamW specifically #
When parameters are bf16 and there's no fp32 master copy underneath, AdamW has three problems:
- Weight decay is a no-op. Decay multiplies each weight by (1 − lr·wd), a relative change of about 1e-9 at these settings. It rounds away on every weight, every step.
- Adam's second moment freezes. The moments are stored in the parameter dtype. At β₂ = 0.999 the 0.1% decay rounds away, so v can never shrink, and once it has grown to about half its true value its increments round away too. This kicks in a few hundred steps in, so it doesn't affect the 100-step run below, but it affects every normal fine-tune.
- Updates smaller than half a gap are dropped. This is the one this article is about.
To be clear, these are known issues. Collage (ICML 2024) worked; the Hugging Face post "Dude, Where's My Update?" (Sep 2026) and a robotics paper from two weeks ago (12 VLA codebases) show it. What we're exploring is what it does to the training process.
3. bf16 is acting as a weight mask #
The survival rule is |Δ| ≥ |w|/512 (roughly). This means a weight can move only if |w| ≲ 512·|Δ|. Adam's step is at most about the learning rate. So at learning rate η, only weights smaller than about 512·η can change, and we can find them statically in pretrained models. We take a look at Mistral-7B below.
bf16's gaps only change size at powers of two, so the cutoff is 512·η rounded down to a power of two. At η = 1e-6 that is 2⁻¹¹ ≈ 4.88e-4, not 5.12e-4. Every weight between 4.88e-4 and 9.77e-4 has a half-gap of about 1.9e-6, bigger than the step, so all of them are frozen.
RMSNorm gains (around 2.2) never move at any learning rate up to 1e-4.
And it really behaves like a fixed mask during training. In our rerun of the Quiet-STaR recipe (section 5), about 1 to 3% of weights change on any given step, but the cumulative set of weights that have ever changed saturates at 5.6%. The same small set gets updated repeatedly.
4. You can see it in published models #
Quiet-STaR (Zelikman et al., COLM 2024) teaches a language model to generate internal "thoughts" before predicting each token. The paper fine-tunes Mistral-7B at learning rate 1e-6, and its released training code does exactly the thing above: bf16 weights, fused torch AdamW, no master copy. The trained checkpoint is on the Hub, so we diffed ezelikman/quietstar-8-ahead against the Mistral-7B-v0.1 it started from.
- 5.4% of the 7.24B weights changed.
- Above 5.12e-4, only 110 of about 6.2 billion weights changed, in any tensor, in any layer. 5.12e-4 is 512 × 1e-6, which is where the rough rule puts the cliff. The exact rule (512 × lr rounded down to a power of two) puts it slightly earlier, at 2⁻¹¹ ≈ 4.88e-4, and that is where the drop is: in the narrow bin just past 2⁻¹¹ (up to 5.10e-4) only 0.39% of weights changed, most likely ones sitting exactly on 2⁻¹¹ that stepped down across it. With an fp32 master copy there is no cliff: small moves accumulate, so the fraction that shows up as changed after casting back to bf16 falls off gradually with weight size instead (Figure 7).
- Below the cliff, about 40% of weights changed, and they moved a lot: half moved by more than 16 bf16 gaps, with a mean relative change of 36%.
| tensor class | share below the cliff | changed, below cliff | changed, above cliff |
|---|---|---|---|
| MLP | 13.6% | 40.4% | 0.000% |
| attention | 14.9% | 36.7% | 0.000% |
| embeddings | 15.4% | 17.5% | 0.000% |
| lm_head | 10.0% | 48.0% | 0.000% |
| RMSNorm | 1.0% | 87.0% | 0.000% |
So the published model is Mistral with its smallest 5% of weights tuned hard and the other 95% untouched.
The new parts tell the same story. The 3-layer mixing head is initialised N(0, 0.02) with a zero output row. Its two hidden layers are still that random matrix, the output row trained to about 3e-6, and its biases sit around 1e-9. The one component that clearly trained is the pair of thought-token embeddings, and that's because the code stores them divided by 100 and multiplies back in the forward pass, a 100× effective learning rate that lands them on the right side of the cliff.
5. Turns out it's a feature, not a bug #
We reran the released recipe exactly as shipped (bf16 Mistral-7B, fused AdamW, lr 1e-6, seed 42, the script's own 100 optimizer steps, one 8×H200 node) and alongside it two corrected versions:
- Shadow : an fp32 copy of the weights that receives the bf16 run's exact clipped gradients through a correct fp32 AdamW. Same gradients, so it isolates what the rounding threw away.
- fp32 master : the whole recipe rerun with fp32 weights, bf16 autocast for the math, and a correct fused AdamW. Same seed, data and everything else.
During the bf16 run, between 97% and 99.7% of each step's proposed updates rounded to zero, and by step 100 it had changed 5.6% of its weights.
Evaluated with the paper's own metric and code. The first two columns are a soft score: the probability the model puts on the correct answer tokens, renormalised over the valid answers. The third is perplexity on held-out OpenWebMath with thoughts switched off.
| GSM8K | CommonsenseQA | perplexity | |
|---|---|---|---|
| Mistral-7B base | 4.96 | 35.72 | 6.04 |
| released bf16 recipe, step 100 | 7.75 | 44.25 | 8.37 |
| shadow: same gradients, correct AdamW | 0.26 | 0.00 | 1,800,000 |
| fp32 master, same recipe | 1.59 | 7.94 | 13.6 |
The bf16 run does what the paper says: both scores go up. Both corrected versions break.
The shadow ends up answering every question with a newline, even though none of its weights is more than one bf16 gap from the base model. A correct optimizer just moved all 7 billion of them a little.
The fp32 run blows up (perplexity 5 × 10¹¹ at step 50, newlines everywhere) and only partly recovers by step 100.
So the published recipe is only stable because of the bug. Learning rate 1e-6 is a setting for an optimizer that discards 99% of its updates.
6. What learning rate did they actually have? #
If bf16 at 1e-6 is secretly a correct run at a smaller step, some lower fp32 learning rate might reproduce it. We did a sweep and found that at 3e-7 a correct optimizer matches the bf16 run's final scores by step 50, within 0.2 points on both tasks, then goes over a cliff (pick "Step 50" in Figure 10 to see it). At step 100, no correct-optimizer run beat bf16: the closest, at 1e-7, scored 41.99 on CommonsenseQA against bf16's 44.25. And bf16 got there by moving a completely different set of weights: 5.6% of them, by a lot, rather than all of them by a little.
Every run that moved the model at all also made the plain language model worse on its own training distribution: bf16 from 6.0 to 8.4, fp32 1e-7 from 6.0 to 7.7 and accelerating. The objective looks unstable on its own, and the rounding bug was holding it together by freezing 95% of the network.
7. The accidental method #
"Only update the small weights" is a published fine-tuning method. So anchoring a fine-tune to its pretrained weights by updating only some of them is a known way to make it stable.
PaFi (Liao, Meng & Monz, ACL 2023) fine-tunes only the 0.5% of weights with the smallest magnitude (plus the normalization layers), using a mask fixed before training, and matches full fine-tuning on GLUE; in their ablation, smallest-magnitude selection beat the alternatives. NanoAdam (Zhou, Jacobs, Gadhikar & Burkholz, NeurIPS 2025) updates only the bottom-k weights by magnitude and re-picks them during training. It beats AdamW on GLUE, nearly eliminates forgetting and, notably, tolerates larger learning rates. Their explanation is that large gradients sit on small weights, while large weights hold pretrained knowledge. Two older methods anchor fine-tuning the same way without looking at magnitude. Child-Tuning (Xu et al., EMNLP 2021) zeroes the gradient of every weight outside a chosen subset; its task-driven version fixes that subset once, before training, by Fisher information. Structurally that is the bf16 bug, with importance in place of magnitude as the selection rule. Mixout (Lee, Cho & Kang, ICLR 2020) randomly swaps weights back to their pretrained values during fine-tuning, which acts as a pull toward the pretrained model and cut the number of failed BERT-large fine-tuning runs.
None of these papers mentions bf16. Meanwhile the RL community has noticed that RL fine-tuning updates look sparse (Mukherjee et al., NeurIPS 2025) and later traced much of that to bf16 rounding (Miahi & Belilovsky, 2026, who derive the same magnitude cutoff; Zhu et al., "The Path Not Taken"). They treat it as a measurement artifact or a bandwidth trick. And the "Dude, Where's My Update?" post saw the same reversal we did on a small Mamba model: at its higher learning rate fp32 master weights did worse, at lower ones better, mechanism "open".
In summary, bf16 round-to-nearest is NanoAdam with the density set by your learning rate. This explains why bf16 recipes tolerate learning rates that blow up a correct optimizer, why the published Quiet-STaR checkpoint has a magnitude cliff, why RL updates look sparse, and why the blog post's ranking flips with learning rate.
If that's right, the rounding itself shouldn't matter, only which weights are allowed to move. To test this we reran the Quiet-STaR recipe five more ways, each changing one thing. The table says what each run was, what we expected, and how it scored. Figure 11 below shows the same runs' training loss.
| run | what we changed | expected, if the mask is what matters | what happened |
|---|---|---|---|
| M1: bf16's mask, exact updates | fp32 weights, but only the weights bf16 would have updated get updated | ≈ bf16 | Stable. Slightly smaller gains (CSQA 41.1 vs 44.3), much less perplexity damage (6.6 vs 8.4) |
| M2: small weights only | fp32 weights; only weights with |w| < 5.12e-4 train, chosen once at the start (like PaFi) | ≈ bf16 | Stable. CSQA 41.8, perplexity 11.7 |
| R: random weights | as many weights as M2 in each tensor, chosen at random | worse than M2, closer to fp32 | Wrong. Stable, as good as M2 (CSQA 44.5), less perplexity damage (9.7) |
| N: NanoAdam | like M2, but the smallest weights are re-picked every 10 steps | ≈ M2 | Right. Each re-pick kept 99.7–99.9% of the same weights |
| T: no thoughts | the bf16 recipe with Quiet-STaR's thought generation turned off | shows whether the gains come from Quiet-STaR at all | No gain (CSQA 34.5 vs base 35.7), so the gains need the thought objective |
All at lr 1e-6, 100 steps, one seed. M2, R and N train 13.8% of the weights, more than twice bf16's 5.7%, so we reran at bf16's exact footprint with three seeds: the smallest 5.7% by |w| against a random 5.7%. Small weights beat random on GSM8K in every seed (7.40 ± 0.25 vs 6.52 ± 0.16), and random did less damage to perplexity (6.70 vs 7.46). CommonsenseQA at step 100 is too noisy across seeds to call.
So the sparsity, not the rounding, keeps the recipe stable: every mask we tried survived a learning rate where dense fp32 blew up. Which weights you pick trades benchmark gain against damage to the language model.
8. What you can do about it #
Check your own runs. The whole diagnostic, run once after warmup at full learning rate:
before = {n: p.detach().clone() for n, p in model.named_parameters()}
loss.backward()
optimizer.step()
changed = sum((p != before[n]).sum().item() for n, p in model.named_parameters())
total = sum(p.numel() for p in model.parameters())
print(f"{changed / total:.1%} of weights changed this step")
If that prints a single-digit number, your optimizer is a sparse fine-tuner.
If you're reproducing a bf16 recipe on a correct stack (DeepSpeed, FSDP mixed precision, fp32 + autocast), don't trust its learning rate. Our data says divide by 3 to 10, and verify stability.
If you want the fix: fp32 master weights. If you can't afford them and must keep weights in bf16, use stochastic rounding. Normal rounding always sends a value to the nearest grid point, so a too-small step is undone every time. Stochastic rounding picks up or down at random, with odds set by how close the value is to each neighbour. In the toy float from §1, w₁ = 4 and Adam proposes 4.05: nearest rounding returns 4 forever, while stochastic rounding returns 5 one time in 20 and 4 otherwise, which averages out to the 0.05 per step Adam asked for. Small updates get paid out occasionally in whole gaps instead of being thrown away. torchao's bf16 AdamW does this (bf16_stochastic_round=True). It fixes the dropped updates and the weight decay, but not the frozen second moment. (If you know zoneout (Krueger et al., ICLR 2017), which randomly keeps an RNN unit's previous value instead of updating it: the bug is a fixed keep-the-old-value mask chosen by weight size, and stochastic rounding turns it into a random one whose odds are set by the step size, so it averages out.)
If you want the accident on purpose: use a sparse update mask and choose the density deliberately. Every mask we tried kept the recipe stable. A small-weight mask gains a little more on benchmarks; a random one does less damage to perplexity. One model, 100 steps, three seeds, so test it on yours.
And torch.optim.AdamW should probably warn when it's handed bf16 parameters. The fused path already inspects dtypes.
Bibliography #
- Yu et al. Collage: Light-Weight Low-Precision Strategy for LLM Training . ICML 2024. §4.1 and App. D.
- juiceb0xc0de. "Dude, Where's My Update?" Hugging Face blog, Sep 2026.
- Singh. Full-Weight VLA Fine-Tuning Without an fp32 Master Copy Silently Erases Most Updates . arXiv 2610.31859, Oct 2026.
- Zamirai, Zhang, Aberger, De Sa. Revisiting BFloat16 Training . arXiv 2010.06192.
- Liao, Meng, Monz. Parameter-Efficient Fine-Tuning without Introducing New Latency (PaFi). ACL 2023.
- Zhou, Jacobs, Gadhikar, Burkholz. Pay Attention to Small Weights (NanoAdam). NeurIPS 2025.
- Xu, Luo, Zhang, Tan, Chang, Huang, Huang. Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (Child-Tuning). EMNLP 2021.
- Lee, Cho, Kang. Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models . ICLR 2020.
- Krueger, Maharaj, Kramár, Pezeshki, Ballas, Ke, Goyal, Bengio, Courville, Pal. Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations . ICLR 2017.
- Mukherjee, Yuan, Hakkani-Tür, Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models . NeurIPS 2025.
- Miahi, Belilovsky. Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL . arXiv 2602.03839.
- Zhu et al. The Path Not Taken: RLVR Provably Learns Off the Principals . arXiv 2511.08567.
- Mukherjee, Yuan, Jayasinha, Hakkani-Tür, Peng. Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs . arXiv 2602.07729, preprint.
- Zelikman et al. Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking . COLM 2024. Checkpoint:ezelikman/quietstar-8-ahead .