{"slug": "non-parametric-rmsnorm-with-normuon", "title": "Non-parametric Rmsnorm with Normuon", "summary": "A paired exploratory ablation on a three-block byte-level Transformer found that keeping the learned per-channel RMSNorm gain (gamma) enabled produced lower validation bits per byte in all six AdamW comparisons and five of six NorMuon-style comparisons, across 24/24 completed arms trained for 1,200 updates each. The probe changed only the six pre-QKV and pre-FFN RMSNorm gamma vectors, with seeds 17, 41 and 79 crossed with gradient-clip thresholds of 1.0 and 3.0, and the author cautioned that the NorMuon effect varied more by seed and clip setting and that the result does not establish gamma-free RMSNorm as a bad design. The author also noted no published or unpublished ablation exactly matching the gamma-only contract was verified, and recommended running a gamma-ON reference against an otherwise identical gamma-OFF run within the optimizer that matters for a given project.", "body_md": "For now, I tried a small experiment:\n\nI take your question to be about one precise intervention: retaining RMSNorm immediately before QKV and FFN projections, but switching its learned per-channel gain (`elementwise_affine`, often called gamma) ON or OFF, while leaving norm locations and the rest of the architecture unchanged. That is a useful distinction. It is not a LayerNorm-versus-RMSNorm test, a pre-norm-versus-post-norm test, or an experiment about adding or removing the final output norm.\n\nI have **not verified a published or unpublished ablation that exactly satisfies that contract** among the sources I checked. I would not infer that none exists; the nearby papers and implementations below are useful for the mechanism or for architectural precedent, but they are not direct substitutes for a gamma-only comparison.\n\nI ran a small, paired, exploratory comparison on a three-block byte-level Transformer. The intervention was limited to the six RMSNorm gains at the pre-attention QKV and pre-FFN locations; a separate final RMSNorm remained non-affine in **every** condition. The hidden 2D matrices used either AdamW or a custom FP32 NorMuon-style update, with gamma, embeddings and the output head on AdamW. I used three seeds and two global gradient-clip thresholds, for **24/24 completed arms**, each trained for 1,200 updates.\n\nAt the final checkpoint, **gamma ON had lower validation bits per byte (BPB) in all six AdamW paired comparisons and in five of six NorMuon-style comparisons**, but the NorMuon effect varied much more by seed and clip setting. These are *three seeds crossed with two clip settings*, not six independent training seeds. In this one toy setting, removing gamma was **not automatically loss-neutral at a matched update budget**. That does not mean gamma-free RMSNorm is a bad design, nor that learned gamma is universally better.\n\nThe main conceptual point remains promising: when gamma feeds directly into a linear projection, `W diag(gamma)` can be represented by a single effective matrix. That is a statement about the **forward function**. It is not an assertion that optimizing `W` and gamma separately, possibly with different optimizers, traces the same path as optimizing the folded matrix alone.\n\nI would keep the gamma-free design on the table, especially if the goal is a cleaner matrix-only Muon/NorMuon parameter organization. Use **a gamma-ON reference and an otherwise identical gamma-OFF run within the optimizer that matters for your project**. Match the seed, common initial weights, sample order, evaluation windows, clipping definition, schedule and training budget. Then choose according to the engineering objective:\n\nNone of those branches requires deciding in advance that one parameterization must win. The point of a controlled ablation is to see where each one works, and at what cost.\n\n| Factor | This probe | \n|---|---|\n| Model | Causal decoder, 3 blocks, width 128, 4 heads, context 96, byte vocabulary 256 | \n| Changed | Six pre-QKV/pre-FFN RMSNorm gamma vectors, all ON or all OFF; `eps=1e-6` | \n| Unchanged | The separate final RMSNorm is **always affine=False** ; same attention/FFN topology, no projection biases or dropout | \n| Data | Fixed contiguous 90/10 split of [Tiny Shakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) | \n| Pairing | Seeds **17, 41, 79** ; matched non-gamma initialized weights, train sample order and validation windows within each seed | \n| Conditions | Hidden 2D matrices on AdamW or NorMuon-style update; global clip at **1.0** or**3.0** ; gamma ON/OFF | \n| Budget | 24 arms, 1,200 steps each, batch 16, context 96; validation at step 0 and every 100 steps | \n| Metric | Next-byte held-out cross-entropy divided by ln(2), in **bits per byte (BPB)** ; lower is better | \n| Precision and optimizer | T4, full FP32, custom Newton–Schulz NorMuon-style update; **not** an unmodified official NorMuon run | \n\nThat is **1,843,200 sampled token positions per arm** (`1200 × 16 × 96`), not unique corpus positions; the same data windows can repeat. Validation uses **9,216 sampled positions per checkpoint per seed** (`6 × 16 × 96`). The windows are shared for ON/OFF comparisons *within* a seed but differ between seeds. Hidden-matrix learning rates were **0.001 for AdamW and 0.02 for NorMuon-style**; auxiliary AdamW LR was 0.001. These are paired parameterization tests, **not** a fairly tuned ranking of the two optimizers.\n\nThe quantity is **BPB(gamma ON) minus BPB(gamma OFF)** for the same seed, matrix optimizer and clip threshold. Negative means ON achieved *lower* BPB. Values are **mean ± sample standard deviation over three seed pairs**, not confidence intervals.\n\n| Hidden-matrix optimizer | Clip threshold | ON minus OFF validation BPB | Sign across 3 seeds | \n|---|---|---|---|\n| AdamW | 1.0 | **-0.021321 ± 0.005683** | ON lower, 3/3 | \n| AdamW | 3.0 | **-0.018895 ± 0.002949** | ON lower, 3/3 | \n| NorMuon-style | 1.0 | **-0.005642 ± 0.015067** | ON lower, 2/3 | \n| NorMuon-style | 3.0 | **-0.017173 ± 0.021250** | ON lower, 3/3 | \n\nThus the two clip cells give AdamW 6/6 and NorMuon 5/6 negative signs, **not** six independent seed replications for each optimizer. One NorMuon clip-1 seed actually preferred gamma OFF. The NorMuon sample SDs are substantial compared with the means, and the checkpoint-time signs can change. Reporting only the aggregate mean, or calling the interaction statistically significant, would be stronger than the data support.\n\n**1. Final validation loss or downstream quality.** The toy **final validation BPB** result is above. At this fixed endpoint, gamma ON was usually lower, most consistently under AdamW in this setup. But there was **no downstream evaluation**: no instruction following, reasoning, multilingual, generation-quality, long-context or separate task benchmark. A difference of hundredths of a BPB on one tiny character-like byte corpus cannot be translated into downstream improvement. Nor have I found a larger external gamma-only result that resolves that question.\n\n**2. Convergence speed at matched compute.** I logged held-out BPB from initialization through step 1,200 at 100-step intervals. Early paired differences were small; the AdamW ON advantage generally grew by later checkpoints, whereas the NorMuon contrasts crossed zero at some intermediate points. This is **matched update count, context and sampled positions**, not measured FLOPs, optimizer overhead, throughput or wall-clock equivalence. Nor were independent LR and schedule sweeps performed. A strict matched-compute claim requires loss-versus-measured-compute curves, and a fair optimizer comparison requires appropriate tuning.\n\n**3. Gradient spikes and late oscillations.** All **24/24 arms finished**; there were zero logged nonfinite training losses, pre-clipping gradient norms or validation checkpoints. With clip threshold **3.0**, there were **0 clipped steps out of 14,400** across both optimizer branches. With threshold **1.0**, clipping affected **54/7,200** AdamW steps and **76/7,200** NorMuon steps, with the *latest* event at steps 11 and 15 respectively. There was **no clipping in steps 601–1,200**. Some NorMuon seed-17 arms ended slightly above their best step-1,100 validation BPB, but that observation alone is not proof of unstable training or late spikes. A longer horizon and an explicit event definition would be needed.\n\n**4. AdamW versus Muon/NorMuon.** The gamma contrast looked more consistent in the AdamW branch and more variable in the NorMuon-style branch. The comparison that is controlled is **gamma ON versus OFF within a fixed optimizer/clip/seed**, not absolute AdamW versus NorMuon BPB. The hidden learning rates differ and my NorMuon matrix update is a **custom FP32 adaptation** for the T4, not the upstream package used unchanged. The [official NorMuon repository](https://github.com/zichongli5/NorMuon) recommends routing hidden matrices to the matrix optimizer and gains, biases, embeddings and head parameters to AdamW. That separate routing is precisely why the gamma question is interesting.\n\nThis is **one three-block toy model, one byte-level corpus, three seeds, 1,200 updates, and a small fixed validation window per seed**. The clip settings reuse seeds and are not independent replications. The extra gamma parameters and their AdamW states are deliberately part of the architecture intervention; identical total parameter counts are not claimed. No strict FLOP matching or downstream metrics were measured, and no production-scale convergence or stability law follows. The NorMuon-specific part was adapted to FP32. An earlier preliminary short run used a runner whose exact bytes are unavailable for a parity audit, so I do **not** combine its numbers or infer why its clipping behavior differed.\n\nThe useful narrow result is that **functional absorbability does not guarantee loss-neutral training at a fixed budget**. It is still reasonable to test an affine-free design on the actual target architecture and optimizer. The details below separate the exact intervention, individual measurements, optimizer reasoning, related work and practical branches.\n\nI would keep affine-free pre-QKV/pre-FFN RMSNorm as a valid research branch. Gamma can be folded into directly following linear projections at the function level, but that does not make separately optimized gamma training-equivalent to a folded matrix. In this specific short, three-seed toy experiment, gamma ON usually gave lower final BPB; the NorMuon-style contrasts were less consistent, and no strict compute, downstream or long-run stability conclusion follows. A matched test on the intended architecture and optimizer is the sensible deciding evidence.", "url": "https://wpnews.pro/news/non-parametric-rmsnorm-with-normuon", "canonical_source": "https://discuss.huggingface.co/t/non-parametric-rmsnorm-with-normuon/191406#post_2", "published_at": "2026-11-04 08:02:56+00:00", "updated_at": "2026-11-04 08:22:10.079323+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "large-language-models", "neural-networks"], "entities": ["RMSNorm", "NorMuon", "Muon", "AdamW", "Tiny Shakespeare", "Newton–Schulz"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/non-parametric-rmsnorm-with-normuon", "markdown": "https://wpnews.pro/news/non-parametric-rmsnorm-with-normuon.md", "text": "https://wpnews.pro/news/non-parametric-rmsnorm-with-normuon.txt", "jsonld": "https://wpnews.pro/news/non-parametric-rmsnorm-with-normuon.jsonld"}}