Non-parametric Rmsnorm with Normuon

A paired exploratory ablation on a three-block byte-level Transformer found that keeping the learned per-channel RMSNorm gain (gamma) enabled produced lower validation bits per byte in all six AdamW comparisons and five of six NorMuon-style comparisons, across 24/24 completed arms trained for 1,200 updates each. The probe changed only the six pre-QKV and pre-FFN RMSNorm gamma vectors, with seeds 17, 41 and 79 crossed with gradient-clip thresholds of 1.0 and 3.0, and the author cautioned that the NorMuon effect varied more by seed and clip setting and that the result does not establish gamma-free RMSNorm as a bad design. The author also noted no published or unpublished ablation exactly matching the gamma-only contract was verified, and recommended running a gamma-ON reference against an otherwise identical gamma-OFF run within the optimizer that matters for a given project.

For now, I tried a small experiment: I take your question to be about one precise intervention: retaining RMSNorm immediately before QKV and FFN projections, but switching its learned per-channel gain elementwise affine , often called gamma ON or OFF, while leaving norm locations and the rest of the architecture unchanged. That is a useful distinction. It is not a LayerNorm-versus-RMSNorm test, a pre-norm-versus-post-norm test, or an experiment about adding or removing the final output norm. I have not verified a published or unpublished ablation that exactly satisfies that contract among the sources I checked. I would not infer that none exists; the nearby papers and implementations below are useful for the mechanism or for architectural precedent, but they are not direct substitutes for a gamma-only comparison. I ran a small, paired, exploratory comparison on a three-block byte-level Transformer. The intervention was limited to the six RMSNorm gains at the pre-attention QKV and pre-FFN locations; a separate final RMSNorm remained non-affine in every condition. The hidden 2D matrices used either AdamW or a custom FP32 NorMuon-style update, with gamma, embeddings and the output head on AdamW. I used three seeds and two global gradient-clip thresholds, for 24/24 completed arms , each trained for 1,200 updates. At the final checkpoint, gamma ON had lower validation bits per byte BPB in all six AdamW paired comparisons and in five of six NorMuon-style comparisons , but the NorMuon effect varied much more by seed and clip setting. These are three seeds crossed with two clip settings , not six independent training seeds. In this one toy setting, removing gamma was not automatically loss-neutral at a matched update budget . That does not mean gamma-free RMSNorm is a bad design, nor that learned gamma is universally better. The main conceptual point remains promising: when gamma feeds directly into a linear projection, W diag gamma can be represented by a single effective matrix. That is a statement about the forward function . It is not an assertion that optimizing W and gamma separately, possibly with different optimizers, traces the same path as optimizing the folded matrix alone. I would keep the gamma-free design on the table, especially if the goal is a cleaner matrix-only Muon/NorMuon parameter organization. Use a gamma-ON reference and an otherwise identical gamma-OFF run within the optimizer that matters for your project . Match the seed, common initial weights, sample order, evaluation windows, clipping definition, schedule and training budget. Then choose according to the engineering objective: None of those branches requires deciding in advance that one parameterization must win. The point of a controlled ablation is to see where each one works, and at what cost. | Factor | This probe | |---|---| | Model | Causal decoder, 3 blocks, width 128, 4 heads, context 96, byte vocabulary 256 | | Changed | Six pre-QKV/pre-FFN RMSNorm gamma vectors, all ON or all OFF; eps=1e-6 | | Unchanged | The separate final RMSNorm is always affine=False ; same attention/FFN topology, no projection biases or dropout | | Data | Fixed contiguous 90/10 split of Tiny Shakespeare https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt | | Pairing | Seeds 17, 41, 79 ; matched non-gamma initialized weights, train sample order and validation windows within each seed | | Conditions | Hidden 2D matrices on AdamW or NorMuon-style update; global clip at 1.0 or 3.0 ; gamma ON/OFF | | Budget | 24 arms, 1,200 steps each, batch 16, context 96; validation at step 0 and every 100 steps | | Metric | Next-byte held-out cross-entropy divided by ln 2 , in bits per byte BPB ; lower is better | | Precision and optimizer | T4, full FP32, custom Newton–Schulz NorMuon-style update; not an unmodified official NorMuon run | That is 1,843,200 sampled token positions per arm 1200 × 16 × 96 , not unique corpus positions; the same data windows can repeat. Validation uses 9,216 sampled positions per checkpoint per seed 6 × 16 × 96 . The windows are shared for ON/OFF comparisons within a seed but differ between seeds. Hidden-matrix learning rates were 0.001 for AdamW and 0.02 for NorMuon-style ; auxiliary AdamW LR was 0.001. These are paired parameterization tests, not a fairly tuned ranking of the two optimizers. The quantity is BPB gamma ON minus BPB gamma OFF for the same seed, matrix optimizer and clip threshold. Negative means ON achieved lower BPB. Values are mean ± sample standard deviation over three seed pairs , not confidence intervals. | Hidden-matrix optimizer | Clip threshold | ON minus OFF validation BPB | Sign across 3 seeds | |---|---|---|---| | AdamW | 1.0 | -0.021321 ± 0.005683 | ON lower, 3/3 | | AdamW | 3.0 | -0.018895 ± 0.002949 | ON lower, 3/3 | | NorMuon-style | 1.0 | -0.005642 ± 0.015067 | ON lower, 2/3 | | NorMuon-style | 3.0 | -0.017173 ± 0.021250 | ON lower, 3/3 | Thus the two clip cells give AdamW 6/6 and NorMuon 5/6 negative signs, not six independent seed replications for each optimizer. One NorMuon clip-1 seed actually preferred gamma OFF. The NorMuon sample SDs are substantial compared with the means, and the checkpoint-time signs can change. Reporting only the aggregate mean, or calling the interaction statistically significant, would be stronger than the data support. 1. Final validation loss or downstream quality. The toy final validation BPB result is above. At this fixed endpoint, gamma ON was usually lower, most consistently under AdamW in this setup. But there was no downstream evaluation : no instruction following, reasoning, multilingual, generation-quality, long-context or separate task benchmark. A difference of hundredths of a BPB on one tiny character-like byte corpus cannot be translated into downstream improvement. Nor have I found a larger external gamma-only result that resolves that question. 2. Convergence speed at matched compute. I logged held-out BPB from initialization through step 1,200 at 100-step intervals. Early paired differences were small; the AdamW ON advantage generally grew by later checkpoints, whereas the NorMuon contrasts crossed zero at some intermediate points. This is matched update count, context and sampled positions , not measured FLOPs, optimizer overhead, throughput or wall-clock equivalence. Nor were independent LR and schedule sweeps performed. A strict matched-compute claim requires loss-versus-measured-compute curves, and a fair optimizer comparison requires appropriate tuning. 3. Gradient spikes and late oscillations. All 24/24 arms finished ; there were zero logged nonfinite training losses, pre-clipping gradient norms or validation checkpoints. With clip threshold 3.0 , there were 0 clipped steps out of 14,400 across both optimizer branches. With threshold 1.0 , clipping affected 54/7,200 AdamW steps and 76/7,200 NorMuon steps, with the latest event at steps 11 and 15 respectively. There was no clipping in steps 601–1,200 . Some NorMuon seed-17 arms ended slightly above their best step-1,100 validation BPB, but that observation alone is not proof of unstable training or late spikes. A longer horizon and an explicit event definition would be needed. 4. AdamW versus Muon/NorMuon. The gamma contrast looked more consistent in the AdamW branch and more variable in the NorMuon-style branch. The comparison that is controlled is gamma ON versus OFF within a fixed optimizer/clip/seed , not absolute AdamW versus NorMuon BPB. The hidden learning rates differ and my NorMuon matrix update is a custom FP32 adaptation for the T4, not the upstream package used unchanged. The official NorMuon repository https://github.com/zichongli5/NorMuon recommends routing hidden matrices to the matrix optimizer and gains, biases, embeddings and head parameters to AdamW. That separate routing is precisely why the gamma question is interesting. This is one three-block toy model, one byte-level corpus, three seeds, 1,200 updates, and a small fixed validation window per seed . The clip settings reuse seeds and are not independent replications. The extra gamma parameters and their AdamW states are deliberately part of the architecture intervention; identical total parameter counts are not claimed. No strict FLOP matching or downstream metrics were measured, and no production-scale convergence or stability law follows. The NorMuon-specific part was adapted to FP32. An earlier preliminary short run used a runner whose exact bytes are unavailable for a parity audit, so I do not combine its numbers or infer why its clipping behavior differed. The useful narrow result is that functional absorbability does not guarantee loss-neutral training at a fixed budget . It is still reasonable to test an affine-free design on the actual target architecture and optimizer. The details below separate the exact intervention, individual measurements, optimizer reasoning, related work and practical branches. I would keep affine-free pre-QKV/pre-FFN RMSNorm as a valid research branch. Gamma can be folded into directly following linear projections at the function level, but that does not make separately optimized gamma training-equivalent to a folded matrix. In this specific short, three-seed toy experiment, gamma ON usually gave lower final BPB; the NorMuon-style contrasts were less consistent, and no strict compute, downstream or long-run stability conclusion follows. A matched test on the intended architecture and optimizer is the sensible deciding evidence.