In Post 2 we built softmax regression from scratch, in Post 13 we let autograd compute our gradients, and in Post 14 we added regularization to keep the weights honest. Every one of those posts quietly assumed the same thing: that once we had a gradient, we knew what to do with it. This post is about that assumption. We will take one fixed softmax regression on MNIST, the handwritten-digit image dataset, and run it through the main first-order and adaptive optimizers, from plain gradient descent to AdamW, and we will watch the same model land anywhere from 41.1 percent to 90.7 percent accuracy depending only on how we move the weights. Think of training as a hiker descending a foggy mountain: the hiker cannot see the valley and only feels the slope underfoot, and every method in this post is a different strategy for choosing the next step.
We keep the task deliberately small so the optimizer is the only variable. We flatten each 28x28 digit into 784 features, hold out 1000 test images, and train on a fixed 6000-image subset with seed 42 so every run is comparable. The model is a single nn.Linear(784, 10), and the loss is cross-entropy, which measures how much predicted probability mass sits on the wrong class. That is the whole setup. Everything that follows is about the step.
What the data tells us #
Before touching an optimizer, we look at the pixels. The train set holds 60000 images at 44.9 MB and the test set 10000 at 7.5 MB, all dense grayscale with zero missing pixels. The label counts run from 5421 for the digit 5 up to 6742 for the digit 1, which puts the majority-class base rate at 0.112. That number matters more than it looks: any accuracy below 11.2 percent is worse than guessing the most common digit, and we will use it as the floor for every table that follows.
Five raw MNIST images: strokes range from clean to smudged.
The five raw samples above show the range we are working with, from clean strokes to smudged ones. The pixel histogram is bimodal, piled up near 0 and 255, so MinMax scaling to [0,1], which simply divides pixel values by 255, is all the preprocessing we need. A duplicate scan over MD5 hashes, checksums that flag identical files, of the flattened images finds 60000 unique train images and zero exact train/test overlap, so we keep the data as is. The hard cases are the low-contrast digits, and they will account for some of the residual error a linear model cannot fix.
Lowest-contrast MNIST samples: faint strokes where a linear boundary will struggle.
With the data understood, we can turn to the step itself.
First-order methods #
The first-order family trades update quality for update count. Gradient descent computes the gradient over the full batch and steps against it, which is stable and uses every example, but on 6000 images it takes only one update per epoch, so three epochs means three steps. Stochastic gradient descent flips that: one sample, one step, thousands of noisy updates. Mini-batch gradient descent averages a small batch and is what everyone actually uses.
Momentum and Nesterov acceleration refine that basic step. Momentum accumulates a velocity so the zig-zag cancels out, and Nesterov acceleration looks ahead along that velocity before measuring the gradient.
for name, bs in [("gd", len(X_train)), ("sgd", 1), ("mini-batch", 128)]:
opt_name = "sgd" if name == "mini-batch" else name
_, losses, acc, _ = train_linear(X_train, y_train, X_test, y_test,
opt_name, epochs=3, batch_size=bs, lr=0.05)
print(f"{name:12s} batch={bs:5d} acc={acc:.3f}")
The results are stark. Full-batch gradient descent reaches 0.411, barely above the 0.112 base rate, because three updates is not enough to move 7840 weights. Stochastic gradient descent hits 0.876, mini-batch lands at 0.852, and adding momentum or Nesterov brings the mini-batch runs to 0.867 and 0.876. The lesson is not that full-batch is wrong, it is that the number of updates matters more than the quality of each one. The hiker who takes three careful steps covers less ground than the one who takes a thousand small ones.
Adaptive methods #
First-order methods share one learning rate across every parameter, which is a poor fit when some weights see large gradients and others barely move. AdaGrad divides each update by the square root of its accumulated squared gradients, so frequently updated weights slow down. RMSProp replaces that growing sum with a decayed average, which fixes AdaGrad's tendency to stall. Adam adds momentum on top of RMSProp, and AdamW decouples Weight decay from the gradient update so regularization is not scaled by the adaptive term. Weight decay shrinks weights toward zero; L2 adds squared weights to the loss.