Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

A new arXiv preprint (2608.23573v1) reports that transformer weight magnitudes follow a two-parameter Weibull distribution with shape k ≈ 1.2, and that the scale parameter λ grows according to a learning-rate-conditioned law λ² - λ₀² = C₀(η) + C₁(η)(H_r - D)^0.59, where D is the bigram conditional entropy computed before training. Across 23 runs spanning an order of magnitude in learning rate, the data collapses onto (H_r - D)^0.59 with unit slope (R² = 0.941), and an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error, making the law a forward predictor of training dynamics.

arXiv:2608.23573v1 Announce Type: new Abstract: A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape $k \approx 1.2$ is stable across layers and models, so the scale $\lambda$ carries most training-induced movement. What corpus property sets how much $\lambda$ grows? Using the bigram conditional entropy $D = H \text{next} \mid \text{prev} $, a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, $\lambda^2 - \lambda 0^2 = C 0 \eta + C 1 \eta H r - D ^{0.59}$, where $H r$ is a matched-budget shuffle baseline. The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per-$\eta$ coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto $ H r - D ^{0.59}$ with unit slope $R^2 = 0.941$; direct per-$\eta$ fits are weaker, $R^2 \approx 0.82$ . Because $D$ is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing. It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader $\Phi D,R,A,H $ data-to-weight framework.