{"slug": "data-predictability-shapes-weibull-weight-scale-growth-in-transformer-training", "title": "Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training", "summary": "A new arXiv preprint (2608.23573v1) reports that transformer weight magnitudes follow a two-parameter Weibull distribution with shape k ≈ 1.2, and that the scale parameter λ grows according to a learning-rate-conditioned law λ² - λ₀² = C₀(η) + C₁(η)(H_r - D)^0.59, where D is the bigram conditional entropy computed before training. Across 23 runs spanning an order of magnitude in learning rate, the data collapses onto (H_r - D)^0.59 with unit slope (R² = 0.941), and an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error, making the law a forward predictor of training dynamics.", "body_md": "arXiv:2608.23573v1 Announce Type: new\nAbstract: A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape $k \\approx 1.2$ is stable across layers and models, so the scale $\\lambda$ carries most training-induced movement. What corpus property sets how much $\\lambda$ grows? Using the bigram conditional entropy $D = H(\\text{next} \\mid \\text{prev})$, a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, $\\lambda^2 - \\lambda_0^2 = C_0(\\eta) + C_1(\\eta)(H_r - D)^{0.59}$, where $H_r$ is a matched-budget shuffle baseline. The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per-$\\eta$ coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto $(H_r - D)^{0.59}$ with unit slope ($R^2 = 0.941$; direct per-$\\eta$ fits are weaker, $R^2 \\approx 0.82$). Because $D$ is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing. It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader $\\Phi(D,R,A,H)$ data-to-weight framework.", "url": "https://wpnews.pro/news/data-predictability-shapes-weibull-weight-scale-growth-in-transformer-training", "canonical_source": "https://arxiv.org/abs/2608.23573", "published_at": "2026-08-26 04:00:00+00:00", "updated_at": "2026-08-26 04:12:44.758870+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/data-predictability-shapes-weibull-weight-scale-growth-in-transformer-training", "markdown": "https://wpnews.pro/news/data-predictability-shapes-weibull-weight-scale-growth-in-transformer-training.md", "text": "https://wpnews.pro/news/data-predictability-shapes-weibull-weight-scale-growth-in-transformer-training.txt", "jsonld": "https://wpnews.pro/news/data-predictability-shapes-weibull-weight-scale-growth-in-transformer-training.jsonld"}}