# Non-parametric Rmsnorm with Normuon > Source: > Published: 2026-11-04 08:02:56+00:00 For now, I tried a small experiment: I take your question to be about one precise intervention: retaining RMSNorm immediately before QKV and FFN projections, but switching its learned per-channel gain (`elementwise_affine`, often called gamma) ON or OFF, while leaving norm locations and the rest of the architecture unchanged. That is a useful distinction. It is not a LayerNorm-versus-RMSNorm test, a pre-norm-versus-post-norm test, or an experiment about adding or removing the final output norm. I have **not verified a published or unpublished ablation that exactly satisfies that contract** among the sources I checked. I would not infer that none exists; the nearby papers and implementations below are useful for the mechanism or for architectural precedent, but they are not direct substitutes for a gamma-only comparison. I ran a small, paired, exploratory comparison on a three-block byte-level Transformer. The intervention was limited to the six RMSNorm gains at the pre-attention QKV and pre-FFN locations; a separate final RMSNorm remained non-affine in **every** condition. The hidden 2D matrices used either AdamW or a custom FP32 NorMuon-style update, with gamma, embeddings and the output head on AdamW. I used three seeds and two global gradient-clip thresholds, for **24/24 completed arms**, each trained for 1,200 updates. At the final checkpoint, **gamma ON had lower validation bits per byte (BPB) in all six AdamW paired comparisons and in five of six NorMuon-style comparisons**, but the NorMuon effect varied much more by seed and clip setting. These are *three seeds crossed with two clip settings*, not six independent training seeds. In this one toy setting, removing gamma was **not automatically loss-neutral at a matched update budget**. That does not mean gamma-free RMSNorm is a bad design, nor that learned gamma is universally better. The main conceptual point remains promising: when gamma feeds directly into a linear projection, `W diag(gamma)` can be represented by a single effective matrix. That is a statement about the **forward function**. It is not an assertion that optimizing `W` and gamma separately, possibly with different optimizers, traces the same path as optimizing the folded matrix alone. I would keep the gamma-free design on the table, especially if the goal is a cleaner matrix-only Muon/NorMuon parameter organization. Use **a gamma-ON reference and an otherwise identical gamma-OFF run within the optimizer that matters for your project**. Match the seed, common initial weights, sample order, evaluation windows, clipping definition, schedule and training budget. Then choose according to the engineering objective: None of those branches requires deciding in advance that one parameterization must win. The point of a controlled ablation is to see where each one works, and at what cost. | Factor | This probe | |---|---| | Model | Causal decoder, 3 blocks, width 128, 4 heads, context 96, byte vocabulary 256 | | Changed | Six pre-QKV/pre-FFN RMSNorm gamma vectors, all ON or all OFF; `eps=1e-6` | | Unchanged | The separate final RMSNorm is **always affine=False** ; same attention/FFN topology, no projection biases or dropout | | Data | Fixed contiguous 90/10 split of [Tiny Shakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) | | Pairing | Seeds **17, 41, 79** ; matched non-gamma initialized weights, train sample order and validation windows within each seed | | Conditions | Hidden 2D matrices on AdamW or NorMuon-style update; global clip at **1.0** or**3.0** ; gamma ON/OFF | | Budget | 24 arms, 1,200 steps each, batch 16, context 96; validation at step 0 and every 100 steps | | Metric | Next-byte held-out cross-entropy divided by ln(2), in **bits per byte (BPB)** ; lower is better | | Precision and optimizer | T4, full FP32, custom Newton–Schulz NorMuon-style update; **not** an unmodified official NorMuon run | That is **1,843,200 sampled token positions per arm** (`1200 × 16 × 96`), not unique corpus positions; the same data windows can repeat. Validation uses **9,216 sampled positions per checkpoint per seed** (`6 × 16 × 96`). The windows are shared for ON/OFF comparisons *within* a seed but differ between seeds. Hidden-matrix learning rates were **0.001 for AdamW and 0.02 for NorMuon-style**; auxiliary AdamW LR was 0.001. These are paired parameterization tests, **not** a fairly tuned ranking of the two optimizers. The quantity is **BPB(gamma ON) minus BPB(gamma OFF)** for the same seed, matrix optimizer and clip threshold. Negative means ON achieved *lower* BPB. Values are **mean ± sample standard deviation over three seed pairs**, not confidence intervals. | Hidden-matrix optimizer | Clip threshold | ON minus OFF validation BPB | Sign across 3 seeds | |---|---|---|---| | AdamW | 1.0 | **-0.021321 ± 0.005683** | ON lower, 3/3 | | AdamW | 3.0 | **-0.018895 ± 0.002949** | ON lower, 3/3 | | NorMuon-style | 1.0 | **-0.005642 ± 0.015067** | ON lower, 2/3 | | NorMuon-style | 3.0 | **-0.017173 ± 0.021250** | ON lower, 3/3 | Thus the two clip cells give AdamW 6/6 and NorMuon 5/6 negative signs, **not** six independent seed replications for each optimizer. One NorMuon clip-1 seed actually preferred gamma OFF. The NorMuon sample SDs are substantial compared with the means, and the checkpoint-time signs can change. Reporting only the aggregate mean, or calling the interaction statistically significant, would be stronger than the data support. **1. Final validation loss or downstream quality.** The toy **final validation BPB** result is above. At this fixed endpoint, gamma ON was usually lower, most consistently under AdamW in this setup. But there was **no downstream evaluation**: no instruction following, reasoning, multilingual, generation-quality, long-context or separate task benchmark. A difference of hundredths of a BPB on one tiny character-like byte corpus cannot be translated into downstream improvement. Nor have I found a larger external gamma-only result that resolves that question. **2. Convergence speed at matched compute.** I logged held-out BPB from initialization through step 1,200 at 100-step intervals. Early paired differences were small; the AdamW ON advantage generally grew by later checkpoints, whereas the NorMuon contrasts crossed zero at some intermediate points. This is **matched update count, context and sampled positions**, not measured FLOPs, optimizer overhead, throughput or wall-clock equivalence. Nor were independent LR and schedule sweeps performed. A strict matched-compute claim requires loss-versus-measured-compute curves, and a fair optimizer comparison requires appropriate tuning. **3. Gradient spikes and late oscillations.** All **24/24 arms finished**; there were zero logged nonfinite training losses, pre-clipping gradient norms or validation checkpoints. With clip threshold **3.0**, there were **0 clipped steps out of 14,400** across both optimizer branches. With threshold **1.0**, clipping affected **54/7,200** AdamW steps and **76/7,200** NorMuon steps, with the *latest* event at steps 11 and 15 respectively. There was **no clipping in steps 601–1,200**. Some NorMuon seed-17 arms ended slightly above their best step-1,100 validation BPB, but that observation alone is not proof of unstable training or late spikes. A longer horizon and an explicit event definition would be needed. **4. AdamW versus Muon/NorMuon.** The gamma contrast looked more consistent in the AdamW branch and more variable in the NorMuon-style branch. The comparison that is controlled is **gamma ON versus OFF within a fixed optimizer/clip/seed**, not absolute AdamW versus NorMuon BPB. The hidden learning rates differ and my NorMuon matrix update is a **custom FP32 adaptation** for the T4, not the upstream package used unchanged. The [official NorMuon repository](https://github.com/zichongli5/NorMuon) recommends routing hidden matrices to the matrix optimizer and gains, biases, embeddings and head parameters to AdamW. That separate routing is precisely why the gamma question is interesting. This is **one three-block toy model, one byte-level corpus, three seeds, 1,200 updates, and a small fixed validation window per seed**. The clip settings reuse seeds and are not independent replications. The extra gamma parameters and their AdamW states are deliberately part of the architecture intervention; identical total parameter counts are not claimed. No strict FLOP matching or downstream metrics were measured, and no production-scale convergence or stability law follows. The NorMuon-specific part was adapted to FP32. An earlier preliminary short run used a runner whose exact bytes are unavailable for a parity audit, so I do **not** combine its numbers or infer why its clipping behavior differed. The useful narrow result is that **functional absorbability does not guarantee loss-neutral training at a fixed budget**. It is still reasonable to test an affine-free design on the actual target architecture and optimizer. The details below separate the exact intervention, individual measurements, optimizer reasoning, related work and practical branches. I would keep affine-free pre-QKV/pre-FFN RMSNorm as a valid research branch. Gamma can be folded into directly following linear projections at the function level, but that does not make separately optimized gamma training-equivalent to a folded matrix. In this specific short, three-seed toy experiment, gamma ON usually gave lower final BPB; the NorMuon-style contrasts were less consistent, and no strict compute, downstream or long-run stability conclusion follows. A matched test on the intended architecture and optimizer is the sensible deciding evidence.