A neural network learns by turning its dials, the weights. Every dial needs a first value. Why not give them all the same one?
Take a small network: 2 inputs, 4 hidden neurons, 1 output. Set every dial to 0.5 and every bias to 0. Now all four hidden neurons see exactly the same thing, so they compute the same value. They make the same mistake, so training gives each one the same push. After the push they are still identical, so the next step repeats it.
We trained this network for 2,000 steps. The four hidden neurons stayed exactly identical the whole time: the difference between them was 0.0 at every step. They are twins, forever.
Every dial starts at 0.5: the four hidden neurons get the same push at every step. Step 2,000, still twins.
Starting every dial at zero is even worse. Then nothing moves at all: every push is exactly zero, the network answers 0.5 for every input, and the loss stays at 0.693 forever (2 of 4 answers right, no better than a coin).
The puzzle has four corners. The output should be 1 when exactly one input is 1: (0,1) and (1,0) give 1, while (0,0) and (1,1) give 0. This is the XOR problem.
Four twin neurons are no better than one neuron, and one neuron can only draw one straight line. As we saw in part 3, one straight line can't solve XOR. The same-start network gets stuck at 3 of 4: its outputs are 0.00, 0.67, 0.67 and 0.67, so the corner (1,1) is wrong, and the loss is stuck near 0.48.
Four twins act like one neuron: one straight split. 3 of 4 right, the top-right corner is wrong.
Now start the dials at random instead (numpy's default_rng(1)). The first-layer dials are [[0.35, 0.82, 0.33, −1.30], [0.91, 0.45, −0.54, 0.58]], and the output dials come from the same generator. Every neuron begins a little different, so each gets its own push and learns its own part. The network gets all 4 corners right from step 18. By step 2,000 its outputs are 0.00, 1.00, 1.00 and 0.00. Its answer is no longer one straight split: it marks the two corners that should be 1 and leaves a clear diagonal band through the other two.
Random start: four different neurons, four different pushes. 4 of 4 from step 18.
Over 20 random starts (seeds 0 to 19), 19 solve the puzzle.
Random isn't enough: the size of the random dials matters too. The CS878 lecture tests this on a deep network: 10 layers of 500 tanh neurons each, fed 1,000 random inputs. Every neuron adds up 500 inputs, each multiplied by its dial.
The right dial size: healthy spreads from layer 1 to layer 10.
A neuron adds up its inputs: s = w₁x₁ + w₂x₂ + … + wₙxₙ. If the dials and the inputs are random with an average of zero, the spreads (variances) add up:
Var(s) = n · Var(w) · Var(x)
To keep the spread the same from one layer to the next, make n · Var(w) = 1, which gives Var(w) = 1/n: dials with a spread of 1/√n. More inputs, smaller starting dials. This is Xavier initialization, named after Xavier Glorot, who worked it out with Yoshua Bengio in 2010. Their paper also balances the backward pass, with Var(w) = 2 / (inputs + outputs). It suits tanh.
ReLU zeroes out half of the values, so ReLU layers need twice as much: Var(w) = 2/n. This is He initialization (Kaiming He and colleagues, 2015).
import torch.nn as nn
layer = nn.Linear(500, 500)
nn.init.xavier_normal_(layer.weight) # tanh layers
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu") # ReLU layers
For a 500-to-500 layer, xavier_normal_ draws dials with a spread of √(2 / (500 + 500)) = 1/√500 ≈ 0.045, the right size from the Short. kaiming_normal_ uses √(2/500) ≈ 0.063. PyTorch already starts every nn.Linear at random, in a range that shrinks as its inputs grow.
The Short's experiment fits in six lines:
import torch as t;n=500
for s in(.01,1,n**-.5):
h=t.randn(1000,n)
for _ in'x'*10:
h=(h@t.randn(n,n)*s).tanh()
print(f'{h.std():.1e}')
It prints the spread after 10 layers for each dial size: 3.0e-07 (fades to nothing), 9.8e-01 (stuck at ±1) and 2.3e-01 (healthy). Swap .tanh() for .relu(): which dial size keeps the signal alive now?
This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.