Gradient descent, the walk every neural network takes while it learns, step by step with real numbers.
Imagine standing in the hills with a blindfold on, trying to reach the bottom of the valley. You can't see the valley or the path. All you have is what you feel under your feet: which way the ground tilts, and how steeply.
That's enough. If the ground tilts down to your left, step left, then feel again. Repeat until the ground is flat. That loop is gradient descent, and it's how almost every neural network is trained.
A network in training is in the same position. Its loss, one number that says how wrong its answers are, depends on its weights: change a weight and the loss goes up or down.
Picture a network with just two weights. For every pair of weights you can compute the loss, so you can draw it as a height over a flat map of the two weights. That gives a landscape. Where you stand on the map is the current weights. The height under your feet is the current loss. Training means finding the lowest point.
In our example the landscape is a long valley:
loss = (w₁ − 3)² + 3 · (w₂ − 1)²
The bottom is at w₁ = 3 and w₂ = 1, where the loss is 0. We start at w₁ = 0 and w₂ = 3, where the loss is 21.
The valley the network can't see. The ball is the current weights; the blue patch is the only ground it can feel.
The network never sees this picture. A real network has millions of weights, so its landscape can't be drawn at all. Like the hiker, it only knows the ground where it stands.
The thing you feel under your feet has a name: the gradient. It's the slope of the loss in every weight's direction at once, and it points uphill, the way the loss grows fastest.
At our starting point the gradient is (−6, 12). The 12 says the loss climbs steeply if weight 2 grows. The −6 says it also climbs if weight 1 shrinks. Going uphill would be a bad idea, so gradient descent does the opposite and steps against the gradient, downhill.
The gradient points uphill (grey arrow). The step goes the other way, downhill (blue).
Each step is the gradient times a small number called the learning rate, written η (eta):
w ← w − η · ∇L
With a learning rate of 0.1, the first step moves the weights by 0.1 × (−6, 12) = (−0.6, 1.2) the other way. They go from (0, 3) to (0.6, 1.8), and the loss drops from 21 to 7.7.
Then you feel the slope again and take the next step. Here is what twenty steps do:
Notice that the steps get shorter by themselves. Near the bottom the ground flattens, so the gradient shrinks, and a small gradient times the learning rate makes a small step. Nobody tells the network to slow down near the minimum. The slope does it.
Twenty real steps at learning rate 0.1. The loss goes from 21 to almost zero, and the steps shrink as the ground flattens.
The gradient gives the direction, and the learning rate decides how far you go each time. To see why that matters, look along the valley floor from the side, where it's a simple curve, loss = (w − 3)², starting at w = 0 with a loss of 9.
The valley floor from the side. At 1.1 every step leaps over the bottom and climbs (grey arcs); at 0.1 the hops shrink and the ball settles at the bottom (blue).
There's a simple reason for the blow-up. On this curve, each step multiplies your distance from the bottom by (1 − 2η). With η = 0.1 that's 0.8, so you get 20% closer every step. With η = 1.1 it's −1.2: you land on the other side, 20% farther away than before.
That's why the learning rate is usually the first setting people tune. A common recipe from the lecture notes: try 0.1, 0.01, 0.001 and 0.0001, and keep the one that makes the loss drop fast within the first hundred steps.
PyTorch can compute the gradient for you. torch.func.grad turns a function into its slope:
from torch import tensor, func
L = lambda w: (w - 3) ** 2
w = tensor(0.)
for _ in range(20):
w -= 0.1 * func.grad(L)(w)
print(w)
It prints tensor(2.9654): twenty steps from 0 get within 0.035 of the bottom at 3. Change 0.1 to 1.1 and run it again to watch the steps leap out of the valley.
This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.