{"slug": "why-does-every-ai-start-out-random", "title": "Why does every AI start out random?", "summary": "A developer's hands-on experiment shows why neural networks must start with random weights: a small XOR network with all weights set to 0.5 kept its four hidden neurons identical for 2,000 training steps, acting like a single neuron and getting stuck at 3 of 4 corners, while zero initialization froze learning entirely with loss pinned at 0.693. Random initialization solved all four corners by step 18, and 19 of 20 random seeds converged. The writeup also walks through Xavier and He initialization, deriving Var(w) = 1/n for tanh layers and 2/n for ReLU layers to keep activation variance stable across a 10-layer, 500-neuron network.", "body_md": "A neural network learns by turning its dials, the weights. Every dial needs a first value. Why not give them all the same one?\n\nTake a small network: 2 inputs, 4 hidden neurons, 1 output. Set every dial to 0.5 and every bias to 0. Now all four hidden neurons see exactly the same thing, so they compute the same value. They make the same mistake, so training gives each one the same push. After the push they are still identical, so the next step repeats it.\n\nWe trained this network for 2,000 steps. The four hidden neurons stayed exactly identical the whole time: the difference between them was 0.0 at every step. They are twins, forever.\n\n*Every dial starts at 0.5: the four hidden neurons get the same push at every step. Step 2,000, still twins.*\n\nStarting every dial at zero is even worse. Then nothing moves at all: every push is exactly zero, the network answers 0.5 for every input, and the loss stays at 0.693 forever (2 of 4 answers right, no better than a coin).\n\nThe puzzle has four corners. The output should be 1 when exactly one input is 1: (0,1) and (1,0) give 1, while (0,0) and (1,1) give 0. This is the XOR problem.\n\nFour twin neurons are no better than one neuron, and one neuron can only draw one straight line. As we saw in part 3, one straight line can't solve XOR. The same-start network gets stuck at 3 of 4: its outputs are 0.00, 0.67, 0.67 and 0.67, so the corner (1,1) is wrong, and the loss is stuck near 0.48.\n\n*Four twins act like one neuron: one straight split. 3 of 4 right, the top-right corner is wrong.*\n\nNow start the dials at random instead (numpy's default_rng(1)). The first-layer dials are [[0.35, 0.82, 0.33, −1.30], [0.91, 0.45, −0.54, 0.58]], and the output dials come from the same generator. Every neuron begins a little different, so each gets its own push and learns its own part. The network gets all 4 corners right from step 18. By step 2,000 its outputs are 0.00, 1.00, 1.00 and 0.00. Its answer is no longer one straight split: it marks the two corners that should be 1 and leaves a clear diagonal band through the other two.\n\n*Random start: four different neurons, four different pushes. 4 of 4 from step 18.*\n\nOver 20 random starts (seeds 0 to 19), 19 solve the puzzle.\n\nRandom isn't enough: the size of the random dials matters too. The CS878 lecture tests this on a deep network: 10 layers of 500 tanh neurons each, fed 1,000 random inputs. Every neuron adds up 500 inputs, each multiplied by its dial.\n\n*The right dial size: healthy spreads from layer 1 to layer 10.*\n\nA neuron adds up its inputs: s = w₁x₁ + w₂x₂ + … + wₙxₙ. If the dials and the inputs are random with an average of zero, the spreads (variances) add up:\n\n`Var(s) = n · Var(w) · Var(x)`\n\nTo keep the spread the same from one layer to the next, make n · Var(w) = 1, which gives Var(w) = 1/n: dials with a spread of 1/√n. More inputs, smaller starting dials. This is **Xavier initialization**, named after Xavier Glorot, who worked it out with Yoshua Bengio in 2010. Their paper also balances the backward pass, with Var(w) = 2 / (inputs + outputs). It suits tanh.\n\nReLU zeroes out half of the values, so ReLU layers need twice as much: Var(w) = 2/n. This is **He initialization** (Kaiming He and colleagues, 2015).\n\n``` python\nimport torch.nn as nn\nlayer = nn.Linear(500, 500)\nnn.init.xavier_normal_(layer.weight)                         # tanh layers\nnn.init.kaiming_normal_(layer.weight, nonlinearity=\"relu\")   # ReLU layers\n```\n\nFor a 500-to-500 layer, xavier_normal_ draws dials with a spread of √(2 / (500 + 500)) = 1/√500 ≈ 0.045, the right size from the Short. kaiming_normal_ uses √(2/500) ≈ 0.063. PyTorch already starts every nn.Linear at random, in a range that shrinks as its inputs grow.\n\nThe Short's experiment fits in six lines:\n\n``` python\nimport torch as t;n=500\nfor s in(.01,1,n**-.5):\n  h=t.randn(1000,n)\n  for _ in'x'*10:\n   h=(h@t.randn(n,n)*s).tanh()\n  print(f'{h.std():.1e}')\n```\n\nIt prints the spread after 10 layers for each dial size: 3.0e-07 (fades to nothing), 9.8e-01 (stuck at ±1) and 2.3e-01 (healthy). Swap .tanh() for .relu(): which dial size keeps the signal alive now?\n\n*This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.*", "url": "https://wpnews.pro/news/why-does-every-ai-start-out-random", "canonical_source": "https://dev.to/logitsolutionsai/why-does-every-ai-start-out-random-ggh", "published_at": "2026-11-04 14:23:34+00:00", "updated_at": "2026-11-04 14:28:34.376080+00:00", "lang": "en", "topics": ["neural-networks", "machine-learning", "artificial-intelligence"], "entities": ["Xavier Glorot", "Yoshua Bengio", "Kaiming He", "PyTorch", "NumPy"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-does-every-ai-start-out-random", "markdown": "https://wpnews.pro/news/why-does-every-ai-start-out-random.md", "text": "https://wpnews.pro/news/why-does-every-ai-start-out-random.txt", "jsonld": "https://wpnews.pro/news/why-does-every-ai-start-out-random.jsonld"}}