TL;DR. Tensor programs are a mathematical language for describing computations in wide neural networks. Their master theorem says that, as the width grows, averages over the neurons become predictable: they converge to expectations in a much simpler scalar random process. This turns the analysis of a high-dimensional random network into a tractable probability calculation and provides a rigorous foundation for studying its infinite-width behavior, including Gaussian-process limits
[Lee et al. (2018)](https://arxiv.org/abs/1711.00165) and neural tangent
kernels [Jacot et al. (2018)](https://arxiv.org/abs/1806.07572).
This note explains the proof for computations that may reuse a random weight matrix and its transpose, as happens with weight sharing and backpropagation. Such reuse creates correlations, so each multiplication cannot simply be replaced by independent Gaussian noise. The proof conditions on all earlier uses of the matrix and separates the next output into a correction determined by those uses and a genuinely fresh Gaussian part. An induction then shows that the scalar process tracks the full computation, including cases in which some intermediate vectors are linearly dependent.
Let and let , independently of . Conditioned on , each coordinate of is Gaussian with variance . This may suggest replacing every matrix product by fresh Gaussian noise, but the next use of the same matrix disproves that idea: The first term converges to . If , the second term, conditioned on , is centered Gaussian with variance
and is independent of . Hence
Thus in fixed-coordinate distribution, not in vector norm. The -term is precisely the dependence that a fresh-noise approximation misses.
The program language, scalar -calculus, and master theorem in this section are from Section 2 of Yang's Tensor Programs III [Yang 2020]. A program starts from random vectors and random matrices , and repeatedly applies
Here and every nonlinearity are fixed as . The matrices are independent, with
and the coordinate slices are iid copies of a fixed Gaussian vector .
The limiting scalar attached to each program vector is defined recursively.
For , use the corresponding component of , and set
For a coordinatewise operation,
Write
For a fixed oriented matrix , the variables are jointly centered Gaussian, with Fresh Gaussian families belonging to different matrices or orientations (in particular, and ) are independent. Reuse through the opposite orientation is restored by
Here the sum is over earlier products on whose fresh Gaussian variables depends. The derivative is symbolic: first express as a deterministic function of all preceding 's, then differentiate that expression.
For nonsmooth functions, the expected derivatives can equivalently be defined by Gaussian regression. If are the relevant earlier products, set Then define the vector of to be . For differentiable functions this agrees with Stein's identity
and its multivariate version.
A function is polynomially bounded if for some . If all program nonlinearities are polynomially bounded, then for any program vectors and any polynomially bounded test function ,
The theorem concerns empirical coordinate laws; it does not assert that the finite-width coordinates are independent.
This is the conditioning step used in the proof of the master theorem (Appendices K and L of
[Yang 2020](https://arxiv.org/abs/2009.10685)).
Order the initial and matrix-generated vectors (the G-vars) as
. Suppose the next one is
where . Collect all earlier uses of in both orientations:
and form
Empty matrices are allowed when or .
Let . Conditioned on , all the displayed vectors are fixed and is subject to Gaussian conditioning under these linear constraints gives the exact law
where is an independent copy of , , , and
Multiplying by yields
with independent of and
This separates the part forced by previous uses of from the remaining Gaussian randomness.
Introduce the finite Gram data
Moment convergence for the preceding G-vars gives deterministic limits, denoted with bars; for example,
Rank stability, proved in the next section, implies and almost surely. Therefore
Indeed, the expression in parentheses is the squared -distance from to .
Using Equation (5), define Then
Consequently, with
we have
Set . By Equation (1), Gaussian regression therefore gives, conditionally on all earlier -variables,
The mean and variance here are exactly and Equation (8).
Applying the ZDot rule, Equation (2), with Stein's identity or its pseudoinverse form, gives the matching drift identity
Adding Equation (11) and Equation (12) proves the key scalar conditional law conditionally on the earlier -variables. For , this reduces to , recovering the opening example.
The core-set and rank-stability mechanism below follows Appendix L of Yang (2020). The proof simultaneously maintains two induction statements.
For every polynomially bounded , There is such that:
The last property upgrades almost-everywhere limiting identities to exact finite-width identities at every coordinate.
To see the consequence, let be polynomially bounded functions of the preceding G-vars and let
If , then Thus the linear combination vanishes almost everywhere under the core law. The density and null-avoidance properties make it vanish at every finite-width coordinate for all large , so . Ordinary lower semicontinuity of rank gives the reverse inclusion. Therefore, eventually,
The pseudoinverse is continuous on a fixed-rank stratum, proving the pseudoinverse convergence used above.
Return to . By Equation (8), There are two cases.
If , fixed coefficients satisfy Expressing and each as coordinatewise functions of the old core turns this into an almost-everywhere functional identity. Core density and null avoidance make it exact at finite width:
almost surely for all sufficiently large . Hence is already in the old span and the core need not change.
If , Equation (13) has the form Conditioned on the old core, has an everywhere-positive Gaussian density. Fubini's theorem then shows that adjoining preserves the two-way null-set property, so the new core is .
At finite width, . Moreover, rank stability and old-core null avoidance imply
Thus every , conditionally on the past, retains positive Gaussian variance. Applying Fubini to each prescribed null set and then old-core null avoidance proves the new null-avoidance property. The same fresh Gaussian component prevents from lying in the old span. This completes the core-set update in both cases.
The final concentration argument combines the projection estimates and weakly correlated Gaussian strong law from Appendices K and L of Yang (2020). It remains to prove . When , the exact relation above makes a fixed polynomially bounded function of earlier G-vars, so applies directly. Assume henceforth that .
From Equation (6), Let
and define the one-dimensional Gaussian average
The difference in the moment invariant is bounded by , where
Conditioned on , the second term inside braces is the expectation of the first. Although the coordinates of are correlated, the dependence has fixed rank. Indeed, if , then the normalized off-diagonal correlations of satisfy
A strong law for such weakly correlated Gaussian arrays applies. Its required high moments follow from polynomial boundedness, , the convergence of , and . Therefore
The projection diagonal satisfies
Hence at most coordinates have ; their contribution vanishes by Hölder's inequality and the same polynomial moment bounds.
On the remaining coordinates, both variances stay bounded away from zero because . Gaussian convolution smooths even a merely measurable polynomially bounded , with
These derivatives are bounded by polynomial weights in the preceding
coordinates and . Meanwhile, Equation (10), Equation (8), and
[Equation (17)](https://www.lesswrong.com/feed.xml#term-b-replace-finite-parameters-by-their-limits) imply
The mean-value bound followed by Cauchy--Schwarz now gives
Every and is a polynomially bounded coordinatewise function of earlier G-vars. Consequently
is polynomially bounded. Applying , then using the scalar identification in Equation (13), gives
Thus , proving .
For the initial Gaussian vectors, the ordinary strong law proves the moment invariant. Choose a maximal linearly independent subfamily of their limiting Gaussian variables as the first core. Its covariance is positive definite, so its law has an everywhere-positive density. Coordinatewise linear relations and countable null-set avoidance prove the other core properties. The preceding sections establish the simultaneous step
Induction therefore proves the moment statement for all G-vars.
Finally, every program vector is a fixed polynomially bounded coordinatewise function of the complete list of G-vars:
The composition of polynomially bounded functions is polynomially bounded. Applying the G-var moment result to gives Equation (3), completing the proof.