{"slug": "weibull-weight-scale-parameter-evolution-under-adamw-training-dynamics", "title": "Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics", "summary": "Researchers derived a three-force decomposition of AdamW training dynamics explaining Weibull weight-scale parameter evolution in transformers, finding alignment force dominates growth. A spline displacement method recovers alignment force from sparse checkpoints with 92-94% accuracy. The peak weight-scale varies with training-data coherence, suggesting data-dependent growth.", "body_md": "arXiv:2606.19367v1 Announce Type: new\nAbstract: Building on a two-parameter Weibull framework for diagnosing transformer weight distributions, we study why the Weibull weight-scale parameter $\\lambda$ grows, overshoots, and then relaxes during AdamW training. We derive a leading-order three-force decomposition of the squared weight norm from the AdamW update: an alignment force measuring the correlation between weights and the adaptive update direction, an injection force from adaptive step magnitude, and a decay force from decoupled weight decay. On self-trained Pythia-70M models with ground-truth optimizer moments, alignment dominates the rise phase, contributing 88-94% of the absolute force budget across four random seeds and remaining robust to super-weight removal. Near saturation, alignment and decay approach balance, explaining the transition from weight-scale growth to relaxation. These force dynamics directly govern the squared-norm component underlying $\\lambda(t)$; the remaining RMS-to-Weibull reconstruction offset is measurable and decomposes into bridge and integration components, totaling approximately 5-6% in densely sampled regions. To extend the analysis to real models where optimizer moments are unavailable, we introduce a spline displacement method that recovers the alignment force from sparse checkpoints with approximately 92-94% accuracy, about twice the naive two-point baseline. We further observe that the peak value of $\\lambda(t)$ varies with training-data coherence in our experiments, suggesting a data-dependent component of weight-scale growth that we leave to a controlled follow-up study. Code and data are available at https://github.com/tiexinding/NPM-Weibull-public.", "url": "https://wpnews.pro/news/weibull-weight-scale-parameter-evolution-under-adamw-training-dynamics", "canonical_source": "https://arxiv.org/abs/2606.19367", "published_at": "2026-06-19 04:00:00+00:00", "updated_at": "2026-06-19 04:08:15.260241+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "neural-networks", "ai-research"], "entities": ["AdamW", "Pythia-70M", "Weibull", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/weibull-weight-scale-parameter-evolution-under-adamw-training-dynamics", "markdown": "https://wpnews.pro/news/weibull-weight-scale-parameter-evolution-under-adamw-training-dynamics.md", "text": "https://wpnews.pro/news/weibull-weight-scale-parameter-evolution-under-adamw-training-dynamics.txt", "jsonld": "https://wpnews.pro/news/weibull-weight-scale-parameter-evolution-under-adamw-training-dynamics.jsonld"}}