# Deep Learning: Searching for a Function

> Source: <https://fazamhd.com/mental-models/deep-learning/>
> Published: 2026-07-23 16:10:45+00:00

Can you write a program which can recognize whether a photo contains a cat or a dog? How would you go about doing it? You might begin with the face, pointed ears suggest a cat, a long snout suggests a dog. But some dogs have pointed ears, and the cat or the dog may not be facing directly at the camera, they might be partly hidden, different lighting conditions, different sizes and positions in the photo, or seen from a different angle. You’ll come across exceptions to every logic, and every new exception will create more rules, it’s never ending.

Yet we are able to do this very easily. After seeing relatively few examples as children, we can recognize cats and dogs we have never encountered before, and in different situations. But it’s hard for us to describe the exact procedure to do that.

Let’s start with a simpler example. Below is a recognizer that distinguishes between a `1`

and a `7`

drawn on a 7x7 grid of pixels.

We have drawn a crossed `7`

in the input grid. Since the recognizer was only trained on uncrossed `7`

s, the horizontal stroke in the middle makes the model wrongly predict the digit as a `1`

.

Beside the input grid is a matching 7x7 panel showing the model’s 49 **weights**, one for each pixel:

- A bar extending to the
**right** indicates that the pixel favors`7`

. - A bar extending to the
**left** indicates that the pixel favors`1`

. - The
**length** of the bar shows how strongly that pixel influences the prediction.

When you draw a digit, the recognizer adds the weights underneath every active pixel. The direction of the total determines its prediction.

Click **Train 7**. The recognizer is shown the answer, adjusts its weights, and learns to recognize this crossed example. You can then draw your own digits and see where its learned rule succeeds or fails.

“We know more than we can tell.” Michael Polanyi

Object recognition is not the only ability that works this way. We recognize faces, understand speech, and notice sarcasm without knowing the exact sequence of decisions involved. In the [software](/mental-models/software) article, we built computation in the traditional way, a person works out a procedure, expresses it as precise instructions, and a computer executes them. That approach works best when the procedure can be clearly articulated. For decades, programmers tried to hand-write rules for reading text, understanding speech, and identifying objects. The resulting systems accumulated rules and exceptions while remaining brittle outside the cases their authors anticipated.

Supervised deep learning reverses the direction of programming:

```
Traditional programming:    rules + data → answers
Supervised deep learning:   data + answers → fitted function
```

Instead of writing the instructions, we collect examples with known answers, define a mathematical function controlled by adjustable numbers, and search for settings that reproduce those answers. Self-supervised and reinforcement learning obtain their learning signals differently, but the broader pattern is the same: combine data or experience with an objective to fit a function. The discovered program does not appear as readable instructions. Its behavior is distributed across millions or billions of fine-tuned numbers.

ChatGPT, Claude, Gemini, image recognizers, speech systems, and self-driving systems are all variations of this idea. Their architectures, data, and learning signals differ, but the underlying mechanism is the same: choose a flexible family of functions, measure its performance against an objective, and adjust it until useful behavior emerges.

My aim with this article is to give you a clear understanding of the underlying system that powers the AI agents you use, recommendation engines deciding the posts in your social media feeds, speech and vision models, and autonomous driving systems.

## Fitting a Function to Data

Let’s begin with a simple example. Consider a digital thermometer. Inside it, a **thermistor** changes its electrical resistance with temperature. The electronics can measure that signal, but the reading is not yet expressed in Celsius or Fahrenheit. To calibrate the thermometer, we would place it at several known temperatures and record pairs of sensor signals and correct readings. The measurements scatter slightly, but they follow a visible trend. We can propose a straight line with two adjustable numbers:

The number (the **weight**) converts a change in sensor signal into a change in temperature, and (the **bias**) corrects the sensor’s zero offset. *Fitting* the line means choosing and so that the line passes as close to the calibration points as possible. Once that is done, the thermometer can turn a sensor signal it has not seen before into an estimated temperature.

Drag either end of the line toward the calibration points. The left end changes the bias; moving the right end also changes the weight. The dotted reading represents a sensor signal the thermometer did not see during calibration.

This simple example contains the entire deep-learning idea. We chose a **family of candidate functions**, every possible straight line, described by adjustable **parameters**, and . We supplied examples with known answers, selected parameter values that fit them closely, and used the resulting function on a signal it had never seen. That final step is **generalization**.

We could instead choose a curve or a function with many inputs. Deep learning begins when we move on to more complex problems. The candidate family becomes a stack of weighted sums and non-linearities with enough adjustable parameters to express rules as intricate as the appearance of a cat.

## Inventing the Artificial Neuron

Researchers borrowed the idea of this function from the brain, although the resemblance is loose. The historical path began with the McCulloch-Pitts model in 1943 and Frank Rosenblatt’s physical perceptron in 1958. The useful modern abstraction is simply a small mathematical unit with adjustable inputs.

An artificial neuron just performs three basic operations:

-
**Weighted sum**: multiply each input by its weight , add the products, and add a bias .Each weight encodes how much that input matters and in which direction (a negative weight means the input argues

*against*firing); the bias shifts how easily the neuron fires at all. Notice that the line-fitting function is exactly this with a single input. The neuron is line-fitting with more inputs. -
**Activation**: pass through a non-linear** activation function**to produce the output. Rosenblatt used a hard threshold: fire if crosses zero, stay silent otherwise. Modern networks use functions with usable slopes, for reasons we will see in the next section.**Sigmoid** squashes any smoothly into the range to , so the output reads naturally as a graded confidence rather than a binary verdict.**ReLU**(Rectified Linear Unit) outputs : zero for negative sums, the sum unchanged otherwise. It costs almost nothing to compute, and it is a widely used activation function.

**Output**: the activated value, which either*is*the answer or becomes an input to further neurons.

A neuron with two inputs has a clean geometric meaning, and it is worth understanding because everything later builds on top of it. Treat the two inputs as coordinates on a plane. The equation then traces a straight line across that plane, and the neuron outputs high on one side of the line and low on the other. The weights control the line’s angle and the bias slides it back and forth. A neuron is a straight-line boundary to its input space, and training a neuron means positioning that line so it separates the examples of one category from the other.

Turn the dials or drag the line to adjust the weights and bias. Click anywhere on the plane to move the input probe and see its output on the left, or press **Train** and watch the neuron position its own boundary until the filled points separate from the hollow ones.

## the invention of the artificial neuron, 1943-1960

In 1943, the neurophysiologist Warren McCulloch and the self-taught logician Walter Pitts published [ “A Logical Calculus of the Ideas Immanent in Nervous Activity”](https://doi.org/10.1007/BF02478259), a mathematical abstraction of the biological neuron, the cell type from which the brain is wired. A neuron receives signals from many other neurons through connections of varying strength and fires its own signal when the combined stimulation crosses a threshold. McCulloch and Pitts showed that this simple unit, stripped to arithmetic, could compute logical functions. Connecting their model to Alan Turing’s 1936

[, they argued that networks of such units could in principle compute any logical function. Their paper helped establish the idea that networks of simple units, none of them intelligent alone, could compute collectively.](https://doi.org/10.1112/plms/s2-42.1.230)

*“On Computable Numbers”*McCulloch and Pitts had a neuron but no way for it to learn; its weights were fixed by hand. The missing idea arrived in 1949 when the psychologist Donald Hebb proposed that connections between neurons strengthen when they fire together, so experience itself could tune the wiring.

“Cells that fire together, wire together.” Donald Hebb, The Organization of Behavior (1949)

In 1957, Frank Rosenblatt at the Cornell Aeronautical Laboratory proposed a machine which could learn this way; his team built it the following year. The **Mark I Perceptron** began with an artificial retina, a modified camera focused an image onto a 20x20 grid of 400 photocells. Each photocell was a light-sensitive resistor. Light falling on it changed its resistance; an amplifier and relay converted this into a binary electrical signal, active for bright patches and inactive for dark ones. Because each cell occupied a fixed position, these 400 signals preserved the image’s rough spatial pattern. To the machine, this was not a triangle or a letter, but rather 400 electrical facts about where light fell. Like the thermistor in the thermometer, the photocell was a transducer, turning a physical property into a signal a calculating circuit could use.

The Mark I Perceptron was slightly more complex than a single neuron. Its 400 sensory units connected, through fixed random wiring, to 512 **association units**. Each association unit received signals from a randomly selected subset of photocells and activated for some combination of light and dark. The association units then fed into eight response units, whose combined activity formed the machine’s response pattern. Only these final connections were learnable, each stored as a motor-driven potentiometer, a variable resistor like a volume dial. Its setting controlled how strongly an association unit influenced a response unit. During training, the operator supplied the desired response along with each image. The learning circuit compared it with the machine’s actual response, and when they differed, drove motors to turn the relevant potentiometers. When they agreed, the settings remained. After training on many images, the machine’s knowledge existed entirely as the final tuned positions of its dials.

The machine attracted extraordinary press coverage. Following a Navy-sponsored press conference in 1958, the New York Times reported the Navy’s expectation of an instrument that would soon be able to walk, talk, see, write, reproduce itself, and be conscious of its existence. It established an early pattern in AI: public expectations racing far ahead of the underlying mathematics.

The learning circuit compares actual and desired responses, then turns the dials when they differ.

Rosenblatt was not the only one trying to build learning machines. In 1960 at Stanford, Bernard Widrow and Ted Hoff built **ADALINE**, a related machine whose learning rule nudged each weight in proportion to its contribution to the error. Generalized forms of this gradient-based correction still drive training today.

Strip away the camera, wiring, and motors, and what remains is the three-operation neuron above. Its learned knowledge existed as physical dial positions; a modern network stores the same kind of knowledge as numbers in memory.

We now have a decision whose behavior is controlled by weights. The next obstacle is adjusting those weights without trying every possible combination.

## Solving Error by Visualizing It as Geometry

First we need a single number that says how wrong the current function is. This number is the **loss**. Run each training example through the model, compare its output with the known answer, and combine the errors. The simulators on this page use **mean squared error**, the average of . Squaring prevents positive and negative errors from cancelling and punishes large misses more heavily than small ones. A perfect fit has zero loss; every wrong answer raises it.

Real classifiers commonly use **cross-entropy**, which scores the probability assigned to the correct category, but the purpose is identical. Once the loss is defined, learning becomes an optimization problem: find parameter values that make that number smaller.

```
prediction → error → slope → parameter update → better prediction
```

## classification loss: softmax and cross-entropy

Networks that classify digits, objects, or the next word in a sentence need two improvements to work with this approach.

First, the output layer will have one neuron per category instead of a single output, followed by a step called **softmax** which rescales their raw scores into confidences that are all positive and add up to 1 (100%). So the output reads as a probability spread across categories. Softmax does this by exponentiating each raw score (making negative numbers positive and magnifying differences), then dividing each by the total sum.

Second, training uses **cross-entropy loss**. Instead of measuring numeric distance, cross-entropy measures the network’s confidence in the correct answer. Predicting 99% confidence for `7`

yields near-zero loss. But because cross-entropy scales logarithmically (), confidence dropping toward 0% doesn’t just double the loss, it sends it spiking toward infinity.

Let’s try to visualize our search space. Let’s take a network with just two parameters, and . We can represent our parameters and the loss on different axes. Since we have two parameters plus the loss variable, we need a three-dimensional representation. Together they form a loss surface, a landscape of hills and valleys over parameter space where low ground means parameters that fit the data very well. A real network has millions of parameters instead of two, which is impossible to visualize directly. But the idea is the same, somewhere on that surface the altitude is lowest, and the search for good parameters is a search for that low ground.

Testing every location is impossible when we have millions of parameters; the space is far too vast for a brute-force calculation of the loss at every combination. The algorithm that works never surveys the entire landscape. It measures the local slope (more on what “slope” means in a moment) and takes a small step in the steepest downhill direction. Repeating these steps is called **gradient descent**. For the linear model and mean squared error used in our example, the surface is a single smooth bowl with one low region, and every slope points toward it. Non-linear networks produce far more complicated surfaces and offer no general guarantee of reaching the absolute lowest point.

But how do we find that steepest downhill direction when we have millions of parameters?

Imagine standing on a hillside in the dark. You want to walk downhill, but visibility is zero, you can only feel the slope of the ground right under your feet. If you measure how steep the ground is along each axis, say, East-West for one parameter and North-South for another; combining those measurements tells you the exact direction of steepest ascent. In mathematics, this combined direction is called the **gradient**. By stepping in the opposite direction of the gradient, you walk in the steepest downhill direction, which is why we subtract the slope rather than add it when updating parameters during training.

A neural network does the same thing, but it does not find the slope by actually nudging each parameter one by one to see what happens; with millions of parameters, that would take forever. Instead, it computes them directly using calculus. The **derivative** is simply a “nudge ratio” for a single parameter, it measures how much the loss changes relative to a tiny nudge in that specific parameter. For instance, if nudging a parameter up by causes the loss to drop by , that parameter’s nudge ratio (its derivative) is . (In standard notation this is written or , the partial derivative of the loss with respect to that parameter.)

Collect this nudge ratio for every single parameter in the network and you have the **gradient**, the complete local slope of the multi-dimensional terrain. To make a training step, we adjust each parameter using its individual nudge ratio, which collectively slides the entire model in the steepest downhill direction.

The learning rate is our step size, the multiplier that scales down the nudge ratio so we don’t take giant, reckless steps. It changes the length of the step, not the downhill direction the gradient points. It is often one of the first settings a machine-learning researcher tunes: framework defaults provide a starting point, but the best value depends on the optimizer, model, and data. Too small, and training could take forever, requiring thousands of steps to cross ground that a bolder setting covers in ten. Too large, and each step could overshoot the valley, land higher than where it started, and bounce wildly across the landscape without converging.

The simulator below visualizes the loss surface for thermometer calibration. Grab the ball and release it to watch it roll down toward the lowest point, while the calibration line updates in real time to minimize error. You can also adjust the learning rate slider to see how step size affects training.

## reading the two panels

On the left, we have the calibration points and the line defined by the current choice of and (determined directly by the ball’s position on the right). On the right, we have the loss surface built by those data points, the two panels are the same thing seen from different perspectives. Every spot on the terrain is a line through the data, and the altitude is how badly that line misses. Grab the ball and watch the line swing as you drag. When you let go and it starts rolling, watch the line settle into the best fit as the ball finds the lowest point. The preview arrow always shows the next step, pointing in the direction of the gradient and growing with the learning rate. Raise the learning rate into the oscillating region and it bounces across the valley. Push it into unstable territory and it flies off the surface. Then try moving the calibration points and watch the terrain deform under the ball, because the loss surface is nothing but the training data viewed from the parameter space.

Two footnotes to this picture carry a lot of practical weight. First, the learning rate is a hyperparameter, a setting that shapes training but is not itself learned. The weights are learned; the learning rate, network architecture, and duration of training are chosen by the machine-learning researcher, often through experiments guided by prior results. Much of day-to-day deep-learning work involves tuning such hyperparameters. Second, you can now see why Rosenblatt’s hard threshold had to go. An all-or-nothing step function is flat almost everywhere: nudging a weight usually changes nothing, leaving no useful gradient to follow. Smooth activations like sigmoid and tanh, and piecewise ones like ReLU, provide slopes through which error can propagate.

One more refinement separates the naive algorithm from the one actually used in practice. Computing the loss over all training examples before each step is exact, but wasteful when the dataset has millions of examples. In practice, each step uses a small, random mini-batch of examples, perhaps 64 or 128 of them, giving a fast, noisy estimate of the gradient. This is called stochastic gradient descent. “Stochastic” is a mathematical term for random or probabilistic, and the noise is a price worth paying, thousands of approximately right steps in general outperform a handful of expensive, exactly right ones. A full pass through the training data mini-batch by mini-batch is called an epoch, and training runs for as many epochs as the error keeps falling, where the error that matters is measured on data held back from training. The reason for that will be explained in more detail in the section on generalization.

## why mini-batches?

We could theoretically calculate the gradient on the entire dataset at once (batch gradient descent) or on a single example at a time (pure stochastic gradient descent). Instead, we choose a mini-batch as a hardware and algorithmic compromise.

-
**Hardware efficiency:** Modern GPU hardware is designed for parallel grid calculations (matrix multiplication). Processing a single example takes almost as long as processing a batch of 64 or 128 because the GPU’s parallel processors sit idle waiting for data. Loading examples into mini-batches keeps the hardware operating at maximum efficiency. -
**Optimization noise:** The gradient calculated on a small subset is a noisy estimate of the full-dataset gradient. This noise is not purely a cost, it changes the optimization path and can sometimes help training move through flat or poorly conditioned regions. -
**Memory constraints:** A mini-batch bounds the memory a training step needs, allowing training on datasets far too large to fit in the GPU’s memory at once.

Batch sizes such as 32, 64, and 128 are common because regular, hardware-friendly dimensions often use GPU memory and parallel units efficiently. The best size still depends on the model, accelerator, memory budget, and optimization behavior. The diagram holds the learning rate constant and shows the same number of updates to isolate gradient noise; with a suitable learning-rate schedule, all three methods can converge.

Smaller batches make each update cheaper, but their gradient estimates noisier.

One further refinement addresses the shape of the terrain itself. Real loss surfaces are rarely perfect round bowls. A more common shape in deep neural networks is a ravine, steep across one parameter direction and nearly flat along another, and plain gradient descent handles ravines badly, oscillating from wall to wall while barely creeping along the floor. Modern training algorithms therefore use **adaptive optimizers**, most commonly **Adam**, which reshape each step parameter by parameter to damp the oscillation and lengthen the stride. The two corrections it applies are explained below, if you’re curious.

## inside Adam: momentum and adaptive scaling

In a ravine, the steep walls dominate the gradient, so successive raw steps bounce between the walls while the component pointing along the floor stays small. Adam corrects the raw step in two ways. **Momentum** averages recent gradients into the update, so the wall to wall components, which alternate in sign, cancel out, while the along the floor component, which is consistent, accumulates. **Adaptive scaling** divides each parameter’s step by a running estimate of that parameter’s typical gradient magnitude, shrinking steps along the steep directions and enlarging them along the flat ones.

That is the learning algorithm for one simple function. But lowering the loss cannot rescue a model that is incapable of expressing the right answer. That limitation produced the field’s first crisis.

## Solving Non-Linear Problems

A single neuron can only draw a straight line, no matter how perfectly it is trained. In 1969, Marvin Minsky and Seymour Papert published [ Perceptrons](https://mitpress.mit.edu/9780262631112/perceptrons/), a rigorous account of what Rosenblatt’s machines could and could not represent. Its best-known example was

**XOR**, the logical function that outputs 1 only when its two inputs differ. We previously built XOR from logic gates in the

[software](/mental-models/software)article.

If you plot XOR’s four cases on a grid, the `1`

s sit at opposite diagonal corners and the `0`

s at the others. No straight line can separate them. Because a single perceptron can only draw straight lines, it cannot represent XOR. No amount of training can teach a model a function it is mathematically incapable of representing. A machine the press had presented as the embryo of artificial intelligence could not reproduce two basic logic gates wired together.

The book became an influential criticism of perceptrons and contributed to declining enthusiasm for neural networks, but it was not the sole cause of the first **AI winter**. Funding priorities, limited computing power, unmet promises across AI, and the absence of a practical way to train multiple layers all mattered. Neural-network research continued, but moved away from the center of the field for more than a decade.

## minsky, rosenblatt, and the perceptron debate

This episode had a bitter personal dimension as well. Minsky and Rosenblatt had known each other since high school and spent a decade publicly disagreeing about the perceptron’s potential. Two years after *Perceptrons* appeared, Rosenblatt drowned in a sailing accident on his 43rd birthday. He never saw multi-layer descendants of his learning machine become a dominant approach to AI.

The architectural solution was already imaginable: place neurons in layers, sending the raw inputs into a hidden layer whose outputs feed the next. But knowing that a multi-layer network could represent XOR was far easier than making it learn. Rosenblatt’s rule only trained the final layer. No practical method existed for adjusting a hidden weight whose effect on the final error was indirect. Without one, layered networks remained a theoretical possibility.

The simulator below shows the limitation of a single neuron and how a hidden layer overcomes it. On the left, steer a single neuron’s line yourself: drag its ends and try to separate the filled points (`01`

, `10`

) from the hollow ones (`00`

, `11`

). Three out of four is the absolute limit. On the right, **auto** trains a network with two hidden neurons from a fresh random start. Switch to **manual** to tune either hidden neuron’s actual weights and bias; the sliders and draggable boundary ends stay synchronized, and the four activation pairs show the representation those parameters create. The fixed output neuron succeeds when the boundaries become parallel diagonals with the filled cases between them. Then drag the warp slider to watch your chosen weights fold the grid and straighten the final decision into one line in hidden space.

The network solves XOR not by teaching one neuron to bend its line, but by learning an intermediate representation that folds the space until a straight-line solution becomes possible. This is the central advantage of deep learning: hidden layers learn a better way to represent the problem.

## tracing the fold by hand

The solution is small enough for us to follow. The labels `00`

, `01`

, `10`

, and `11`

track the four XOR cases in both panels, and the dashed lines and in the right panel are the straight boundaries of the two hidden neurons. Instead of forcing a single neuron to solve the problem in one step, the network breaks it into a two-step procedure.

**Step 1: Warp the Space (The Hidden Layer).** The hidden neurons act as intermediate detectors. Each hidden neuron draws a straight line partitioning the input space (the dashed lines above). These lines warp the grid, remapping the coordinates of our points from the original space to a new hidden space , based on which side of each line the input lands on.**Step 2: Slice the Space (The Output Neuron).** In this new coordinate space, the points are rearranged to be easily separable. The output neuron simply draws a single straight line in space to separate them.

To see how this works intuitively, imagine each hidden neuron is a specific detector. **Neuron 1** fires only when both inputs are high, and **Neuron 2** fires only when both inputs are low. (The hidden neurons here use **tanh**, a scaled relative of the sigmoid that squashes into the range to instead of to , so a firing neuron outputs about and a quiet one about .)

Mapping the four XOR cases into the new coordinate space:

| Input | XOR Output | Neuron 1 (both-high detector) | Neuron 2 (both-low detector) | Hidden Space |
|---|---|---|---|---|
`01` | 1 | Quiet | Quiet | |
`10` | 1 | Quiet | Quiet | |
`11` | 0 | Fires | Quiet | |
`00` | 0 | Quiet | Fires |

Read the first two rows, the hidden layer collapses the two mixed cases `01`

and `10`

to the exact same spot, , because neither detector fires, while pushing the matching cases `11`

and `00`

to separate corners. Notice how “Fires” lines up with output 0, not 1: the detectors detect sameness, and XOR is true exactly when sameness is absent. That inversion is what makes the output neuron’s job trivial, answer 1 when both detectors stay quiet, a rule one straight line in hidden space can draw.

To visualize this, imagine folding a square sheet of paper along a diagonal. By bringing two opposite corners together to touch, the square becomes a triangle. A single straight cut can now easily slice the folded corner away from the other two. Every solution found by gradient descent performs this same essential move, it folds the input plane so that the points become linearly separable. Depending on the random initialization, it may collapse the `01`

and `10`

pair together, or equivalently collapse `00`

and `11`

instead, but the geometric strategy remains identical. The simulator always picks a run that collapses the mixed pair, so what it plays matches this explanation.

Four points are the smallest possible instance of this approach, but the exact same strategy works on higher-dimensional problems. The simulator below trains a network on handwritten digits, 49 pixel inputs, a hidden layer of just two neurons, and one output. Input space now has 49 dimensions, one per pixel, making it impossible for us to visualize. But the hidden layer compresses every image down to two numbers, mapping it onto a plane that fits on the page. Each training example is drawn as its own glyph at its hidden coordinates . At the start, the glyphs scatter without order, `7`

s and `1`

s mixed together; as training progresses, the `7`

s collect on one side and the `1`

s on the other, until the output neuron’s straight line separates them, exactly as it separated the folded XOR points.

Look closely at the clusters. Crossed `7`

s sit next to plain `7`

s, and `1`

s with serifs land beside bare strokes. The hidden layer preserves the features that distinguish a `7`

from a `1`

, while discarding details it doesn’t need, like handwriting style. Try drawing or editing a digit to watch its point move across this 2D plane. You might see occasional misclassifications, like a `7`

getting classified as a `1`

. That is expected, this model is intentionally minimal, trained on only a handful of examples.

This is what *learning a representation* means, and the same thing happens with more complex tasks. A network reading full photographs performs the exact same steps, but with more layers and millions of dimensions, reshaping its space until the categories separate, preserving only the distinctions its task rewards. Pay close attention to this dependence on what the task rewards, it becomes crucial when we examine generalization.

XOR needed only two hidden neurons. With more hidden neurons, a network can form decision boundaries far more complex than a single straight line. Results proved independently by George Cybenko, Kurt Hornik, Maxwell Stinchcombe, and Halbert White in 1989 established what became known as the **universal approximation theorem**: under suitable conditions, a network with even one sufficiently wide hidden layer can approximate any continuous function to arbitrary precision. Its mathematical toolbox is large enough to represent the unwritable rules, although the theorem does not say that gradient descent will find them, or how much data and width that search will require.

The theorem is a statement about functions, so the cleanest place to watch it is one input and one output, where the network’s entire behavior is a single curve on the page. Draw a target curve below, then change the hidden layer’s width. Each hidden neuron produces an adjustable S-shaped curve, which appears as the faint curves below. Training can move each curve left or right, make its transition sharper or gentler, scale it, or flip it upside down. The network adds the adjusted curves together to produce the final thick fitted curve.

With only one or two neurons, the network has too few pieces to follow every part of the target curve. Adding neurons supplies more pieces, allowing the gaps to close until the combined curve tracks the target function more intricately.

Approximating a complex curve by adding up simpler mathematical pieces is a classic technique, Fourier series do it with sinusoids, and Taylor series do it with powers of . In those classical methods, the pieces are fixed in advance, you can only adjust their weights (how much each piece contributes).

What sets a neural network apart is that its pieces are **flexible**. The network learns not just the weight of each S-curve, but also its position (where it sits on the axis) and steepness (how sharply it rises). This lets training concentrate the pieces exactly where the target function is most complex, rather than spreading them evenly everywhere.

Redraw the target and watch the faint curves migrate toward the wiggles. That adaptivity is what lets the same approach scale to higher dimensions, where covering a high-dimensional space with a fixed grid of templates would require exponentially many pieces, flexible curves can slide together to cover the narrow regions where real data actually lives.

## scaling math: from single neurons to deep layers

A single layer is thousands of neurons side by side, each reading every input from the previous layer, and a deep network stacks dozens of these layers. Collecting layer ‘s weights into a matrix (one row per neuron), the entire 100-layer network is a single nested expression:

where stands for the activation function (such as Sigmoid, ReLU, or ).

At 8,192 neurons per layer each reading 8,192 inputs, a single holds 67 million weights; a hundred such layers hold close to 7 billion parameters. Every one of those numbers is a weight playing the exact same role as in , set by the same fitting procedure.

Adding hidden layers and scaling up parameter counts changes not just what the network can represent, but the geometry of fitting it. While the single line’s loss landscape was a simple **convex** bowl (one that curves smoothly to a single lowest point), the XOR network’s nine parameters create a **non-convex** surface, a bumpy terrain of hills and ridges where walking downhill no longer guarantees reaching the absolute lowest point. This new landscape can fold into flat shelves, saddle points (points that slope up in some directions but down in others, like a mountain pass), and multiple separate valleys.

## why the loss surface has two valleys

These valleys are not hypothetical, and they arise naturally from a **permutation symmetry** in the network. If you take a solution of the XOR network and swap its two hidden neurons (along with their weights), you get a different point in parameter space that computes the exact same function.

Since the function is identical, it has the exact same loss. The loss surface must therefore contain at least two equally deep valleys, one for each neuron ordering. (With activations like , flipping the signs of a neuron’s input weights and its output weight also leaves the output unchanged; permutation symmetry alone guarantees at least two symmetric valleys for two hidden neurons.)

Because we can’t draw nine dimensions, the diagram below cuts a 2D cross-section through the landscape, one direction walks from a known solution to its swapped twin, and the other varies the output bias (receding into depth as increases). Between the twins, the surface rises into a ridge.

At the halfway point between the twins, both hidden neurons end up with the exact same weights. The two neurons collapse into one, making the network unable to solve XOR and raising the loss. The lowest crossing over this ridge is a saddle point.

The two valleys in this slice are equally good: they compute the same function at the exact same loss. But a non-convex surface can also contain poor basins, flat plateaus, and saddles, which brings back the question we postponed. If gradient descent only knows the local slope, why does it usually find a useful solution? No single theorem guarantees success. Instead, what helps in practice is **overparameterization**: having many extra parameters creates redundant dimensions. A point that looks like a trap along one or two directions often has a downward slope in another. In high dimensions, there are many different paths down and many different valleys that fit the data equally well. Optimization can still fail on flat plateaus or unstable steps, but the mere fact that the surface is non-convex does not make training hopeless.

This bumpy geometry was not just a technical inconvenience; it shaped the field’s history for two decades.

## the convex era: the twenty-year SVM detour

Through much of the 1990s and 2000s, **Support Vector Machines (SVMs)** became more attractive than neural networks for many classification problems. The easiest way to see why is to compare an SVM with the XOR network above. That network did two things: it *learned* a warp of the input plane, then drew a straight line in the warped space.

An SVM with a kernel keeps the same broad two-step picture but fixes the first step in advance. The chosen **kernel** defines how similarity between points is measured, corresponding to a fixed representation in which training fits a linear boundary. Unlike a neural network’s hidden layers, that representation is not learned jointly with the classifier from the task loss.

XOR itself shows what a frozen warp looks like: encode the two inputs as and instead of and , and the product is exactly when the inputs match. Add that product as a third coordinate and the matching cases lift above the mixed ones, so a flat sheet separates what no line in the plane could. These were explicit instructions someone had to figure out.

Fixing the representation makes the remaining optimization problem **convex**, any local optimum is also global, so training does not face the many poor local solutions that researchers feared in neural networks. For separable data, an SVM chooses the boundary with the largest margin from the nearest training points on either side; those nearest points “support” the boundary, hence the name.

Given the same formulation and settings, an SVM solver can reliably target a global solution. A neural network trained from two random initializations may settle into different valleys and produce different boundaries. To researchers who had watched networks stall and diverge unpredictably, convex guarantees made SVMs look far more practical. (The wider AI field had meanwhile gone through a second AI winter at the end of the 1980s, when the commercial market for expert systems collapsed; neural networks remained outside the mainstream for years afterward.)

Both panels fit the same points. Left: the SVM's one optimal boundary, identical on every run. Right: three training runs of the same network from different random initializations, three different boundaries.

But the guarantee had a cost. A kernel fixes the representation before classifier training rather than learning it end to end from raw data. If that representation does not expose the distinctions the task needs, the boundary cannot recover them. An SVM therefore has no equivalent of hidden layers learning edges, shapes, and objects together with the final decision. Convexity was obtained by fixing the part of the model that deep learning aims to learn. A non-convex landscape is part of the cost of learning the representation and classifier together; overparameterization and improved optimization techniques helped make that trade practical.

## Backpropagation: Assigning Blame Through a Network

Multiple layers solve the representation problem, but create a new one. For a single neuron, we can directly calculate how each weight affects the loss. A hidden weight acts only indirectly: it changes one neuron’s output, which changes later neurons, which eventually changes the answer. If that answer is wrong, how much blame belongs to each weight along the way? This is the **credit-assignment problem**. Without an efficient solution, gradient descent cannot train the representations that make layers useful.

The breakthrough was **backpropagation**, popularized for neural networks in 1986 by David Rumelhart, Geoffrey Hinton, and Ronald Williams in [ “Learning Representations by Back-propagating Errors”](https://www.nature.com/articles/323533a0). Its lineage reaches further back: Seppo Linnainmaa described reverse-mode automatic differentiation in 1970, and Paul Werbos proposed applying the method to neural networks in 1974. The 1986 paper demonstrated clearly that it could make hidden representations learn in practical networks. Its core is the chain rule of calculus. When a parameter affects the loss through a chain of operations, its total nudge ratio is the product of the local nudge ratios along that path.

A layered network is exactly such a chain. First run it forward to compute the prediction and loss. Then move backward through the recorded operations, passing responsibility from each output to the inputs that produced it. One forward pass and one backward pass, costing only a small multiple of the prediction itself, produce the gradient for every parameter, whether the model contains nine weights or nine billion.

To understand credit assignment with more clarity, isolate a single output neuron. Two incoming activations () combine with weights () and bias into the raw **logit** , which sigmoid turns into a prediction between and . Drag any parameter to see downstream values update, then release it to watch loss flow backward. At the sigmoid, loss is multiplied by the local slope, prediction (1 - prediction); at the sum, the gradient splits unchanged; at each product, it is multiplied by the incoming activation to yield that parameter’s gradient. All parameters then step downhill together, repeating until the prediction settles in the target region.

That completes the modern training loop: **forward pass** (compute outputs), **loss** (measure error), **backward pass** (compute gradients), and **update** (step parameters downhill). Modern deep learning frameworks abstract these details so we can focus on the problem itself. Below is what that loop looks like in PyTorch.

```
for inputs, targets in data:                   # mini-batches
    outputs = model(inputs)                    # forward pass
    loss = ((outputs - targets) ** 2).mean()   # loss (mean squared error)
    loss.backward()                            # backward pass: every gradient
    optimizer.step()                           # update: every parameter downhill
    optimizer.zero_grad()
```

Whether `model`

holds the nine parameters of the XOR network above or billions, these six lines are the whole training loop. Frameworks generate the backward pass automatically by recording the forward computation, a mechanism the PyTorch article will examine in detail. Architectures have changed enormously since 1986, but they still learn through this same cycle.

The simulator below runs that loop on the XOR network from the previous section. Each training example enters the shared model: its prediction flows forward, compares against its known target, and sends a correction backward. The update changes predictions for all four examples because they all share the same nine parameters. The four examples complete an epoch, and the same loop repeats until the hollow cases `00`

and `11`

reach the target region near while the filled cases `01`

and `10`

reach the region near . The first epoch plays slowly enough to read before the repetition accelerates.

## Training Finds the Program, Inference Runs It

Everything so far has described **training**: repeatedly showing the model examples, measuring loss, running backpropagation, and changing the weights. Once training finishes, those weights are saved. Using the resulting model is a different process called **inference**. A new input passes forward through the frozen weights to produce an output; there is no target answer, no loss, no backward pass, and normally no parameter update.

Training may process a dataset many times across thousands of accelerators. A simple inference needs only a forward pass, although a large model may still require substantial memory and computation. This asymmetry is why organizations can spend months creating a model and then serve copies of it millions of times. It is also why your interaction with a deployed model does not ordinarily teach its shared weights immediately: if the interaction is retained and selected, it may later become evaluation or training data, but changing the model requires another controlled training run and deployment.

The distinction creates two different engineering problems. Training asks how to learn better weights reliably and efficiently. Inference asks how to produce outputs with acceptable latency, memory, energy use, and cost. Techniques such as quantization, distillation, caching, and specialized inference chips reduce the second cost without repeating the original search from scratch.

There is one important complication at the current frontier: inference can itself contain a search. A system may sample several possible answers, check them, call tools, or spend additional steps revising a solution while keeping its weights frozen. This is **test-time computation**, more work with the learned program, not further training of the program, and we will return to it near the end.

## Convolutional Neural Networks: Reusing Patterns Across an Image

Backpropagation made hidden layers trainable. Researchers could now shape the network around a problem while keeping the same four steps underneath: forward pass, loss, backward pass, update. An architecture is therefore not a different learning principle; it is a choice about which patterns should be easy for the shared learning loop to discover.

Three years later, in 1989, this system met an important practical test, reading handwritten digits on U.S. mail. In [ “Backpropagation Applied to Handwritten Zip Code Recognition”](https://doi.org/10.1162/neco.1989.1.4.541), Yann LeCun and his Bell Labs colleagues trained a network to solve it from raw pixels, end to end with backpropagation.

The architecture had a history of its own. In 1980, Kunihiko Fukushima published [ “Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position”](https://doi.org/10.1007/BF00344251), describing a layered vision network modeled on Hubel and Wiesel’s physiology of the cat’s visual cortex. The

**neocognitron** alternated local pattern detectors with layers that tolerated small shifts in position. Fukushima had the right wiring, but no general way to train it. LeCun’s insight was to train essentially that architecture with backpropagation.

Alternating feature detection with local pooling builds larger receptive fields while reducing sensitivity to position.

Author’s schematic based on K. Fukushima, [“Neocognitron” (1980)](https://doi.org/10.1007/BF00344251).

The result, the **convolutional neural network** (CNN), built one useful assumption about images into its wiring: a pattern worth detecting in one region, an edge, loop, or stroke, is probably worth detecting everywhere. Instead of assigning separate weights to every position, a convolutional layer learns small pattern detectors and slides them across the image, reusing the same weights at every location. This reduced the parameter count enormously, and LeCun’s system was soon deployed to read handwritten checks and zip codes.

The filter in the simulator below is one such learned pattern detector, reused unchanged at every position. Drag it across the input: its weighted sum appears at the corresponding position in the feature map. Then shift the input pattern and watch the exact same detector find the same stroke somewhere else.

## the vanishing gradient problem

Backpropagation did carry one weakness: invisible at three layers and crippling at ten. The backward pass multiplies the error signal through layer after layer of activation derivatives, and the sigmoid’s derivative is everywhere less than one, so the product shrinks geometrically: by the time the signal reaches the early layers, almost no gradient is left, and those layers barely learn. This **vanishing gradient problem**, diagnosed by Sepp Hochreiter in 1991, is why depth stayed stuck at two or three layers for years. And it is a large part of why ReLU (whose derivative is exactly one across half its range) displaced the sigmoid in modern networks.

The rest of the solution came later and was architectural: **skip connections**, introduced in 2015 by Kaiming He and colleagues in [ “Deep Residual Learning for Image Recognition”](https://arxiv.org/abs/1512.03385). Its residual network (ResNet) used wires that carry a layer’s input around the layer and add it back on the far side, so the error signal always has a path backward that no layers of shrinking derivatives can kill. They are the reason networks today run to hundreds of layers, and why transformers underlying LLMs like ChatGPT and Claude can scale so deep.

## Generalization: What the Network Actually Learns

The training loop minimizes loss *on the examples it receives*. But the useful goal is performance on new inputs, and those two goals can quietly diverge. Searching for a program does not guarantee finding the program we intended, only one that scores well on the world represented by the training data.

A function family flexible enough to represent the true pattern is usually flexible enough to represent much more than that, including the accidental noise in the particular examples collected. Given enough capacity, the fitting procedure will happily spend it threading the function through every quirk, measurement error, and mislabeled point in the training set, driving the training loss toward zero while drifting *away* from the underlying pattern. John von Neumann captured the danger of excessive model capacity in one image:

“With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.”

John von Neumann

This failure is called **overfitting**, and training performance alone cannot reveal it. The ability to perform well on new data is called **generalization**, so we estimate it using examples excluded from training. A final **test set** is split off before model development and kept untouched; its error estimates how the finished system performs on new examples drawn from the same distribution. The gap between training and held-out performance is called the **generalization gap**, although distribution shifts and noisy measurements can make real-world performance differ further.

To keep the test set truly unseen, hyperparameter tuning gets its own third split, a **validation set**. If you tune model settings by watching test performance, you quietly leak test data into your design choices. Beyond collecting more data, practitioners also fight overfitting with **regularization**, techniques like weight decay and dropout that penalize complex fits, trading a little training perfection for better generalization.

Training fits the model. Validation guides design choices. Test estimates final performance.

Overfitting can also reveal itself over time. Training error often continues falling while validation error improves at first, then levels off or begins to rise. This divergence suggests that further fitting is no longer improving generalization. A standard defense called **early stopping** keeps the checkpoint with the best validation performance instead of automatically using the final training step.

You can observe this more clearly in the simulator below. The filled points are noisy training data, hollow points are the held-out test set, and the dashed curve is the true underlying pattern. At low flexibility, the model is too rigid to follow the pattern (underfitting). In the middle, it tracks the pattern and ignores the noise. At maximum flexibility, it hits every training point perfectly, training error hits zero, while test error increases. Drag a point to see how sensitive high-flexibility fits are, or click anywhere to add data and watch more points tame the swings.

The classical bias-variance picture predicts a U-shaped relationship between model capacity and test error: models that are too simple underfit, while models that are too flexible can overfit. Some modern learning systems show a more surprising pattern. Under certain combinations of architecture, data, noise, and training procedure, increasing capacity past the point where the model can fit every training example produces a second drop in test error.

## deep double descent: past the interpolation threshold

In experiments that exhibit **deep double descent**, test error first follows the classical U-curve and peaks near the **interpolation threshold**, where the model has just enough capacity to fit every training point. Increasing capacity further can make test error fall a second time, sometimes below the best error in the underparameterized regime. This behavior is not universal, but it helps explain why parameter count alone is a poor measure of whether a modern model will generalize.

Why can this happen? Overparameterized networks have many ways to fit the training data, and the training algorithm does not choose among them uniformly. Initialization, architecture, optimizer, and step size bias the search toward some solutions over others, an effect called **implicit regularization**. In some settings this bias favors solutions that generalize well despite having enough capacity to memorize the data. The phenomenon remains an active area of research, so no single explanation covers every model and dataset.

To put it simply, the loss specifies what counts as success, and the dataset supplies the world in which success is measured. A neural network has no independent notion of the rule we meant. The statistician George Box’s aphorism applies exactly:

“All models are wrong, but some are useful.”

George Box

Gradient descent finds whatever regularities most easily reduce the loss, with no preference for the *intended* ones. If wolves in the training photos usually stand in snow, detecting snow can lower the loss just as effectively as detecting a wolf. The model has passed the supplied test without learning the intended distinction.

The repair is not merely “more data,” but data chosen to break the accidental correlation. A wolf on grass shows that grass cannot mean dog; a dog in snow shows that snow cannot mean wolf. These **counterexamples** force the fit away from the cheap background shortcut and toward features that survive across backgrounds. In a real system, fixing this requires retraining the network and testing it on fresh, unseen combinations. A high training score alone doesn’t prove the model unlearned the shortcut, only held-out test cases can certify the repair.

If a hiring dataset embeds decades of biased human decisions, fitting that data reproduces those biases. If an input lands far from everything in the training set, the function still returns a value, often with unjustified confidence, because its mathematics extends even into regions no example constrained. This shifts where debugging happens. In traditional software, a bug often lives in readable instructions. In a learned system, the failure may live in the examples, labels, objective, architecture, or evaluation. Debugging means finding which evidence rewarded the wrong behavior and testing whether the fix holds on unseen cases.

There’s also a genuinely encouraging discovery about what the parameters end up containing. When researchers inspect trained vision networks, they find that depth organizes itself into a hierarchy: neurons in early layers become detectors for edges and patches of color; middle layers combine those into textures, corners, and parts; late layers combine those into faces, wheels, and words. Features built from features built from features, none of them designed by anyone. This hierarchy is why deep networks beat wide-but-shallow ones in practice, even though the universal approximation theorem says a shallow network could represent the same functions in principle. The world itself is compositional, objects are made of parts made of textures made of edges, and a network that can reuse intermediate concepts matches that structure exponentially more economically than one that must build every concept directly from raw pixels. These layers were not designed by anyone; gradient descent discovered them because the loss surface rewarded it.

## When Failures Become the Next Dataset

Training a model once is only the beginning of a real deep learning system. The first dataset reflects the situations its creators anticipated; deployment reveals the ones they missed. An unusual road layout, a partly hidden pedestrian, glare across a camera, or an object carried in an unexpected position may expose a failure that was nearly absent from the original data. Each failure identifies evidence the model needs next.

Andrej Karpathy, who led Tesla’s Autopilot vision team from 2017 to 2022, described neural networks as [ Software 2.0](https://karpathy.medium.com/software-2-0-a64152b37c35). The weights behave like a program, but engineers rarely repair that program by editing individual numbers. They change the system that produces the numbers: collect different examples, correct labels, alter the objective or architecture, retrain, and evaluate the resulting model.

Tesla’s driving system made this workflow unusually concrete. A deployed model encounters far more situations than engineers could assemble beforehand. The useful cases are not millions of repetitive clear roads, but rare situations where the current model is wrong or uncertain. Engineers can search recorded fleet data for those cases, label them, add them to the training and evaluation sets, retrain the network, and deploy the next version. The deployment discovers weaknesses; the weaknesses determine the next dataset; the new dataset changes the program.

Karpathy called the machinery around this cycle a **data engine**. Merely owning a large pile of data is not enough. The advantage comes from repeatedly finding which examples would most improve the model, acquiring and labeling them, retraining, checking the new model against both old and new failures, and deploying it safely enough to begin the cycle again.

This changes the meaning of a bug fix. Correcting one `if`

statement in conventional software usually changes one explicit branch. Adding examples and retraining changes a shared statistical function, so a repair in one region can unexpectedly alter behavior elsewhere. Every known failure therefore needs to become a permanent **regression test**, and every candidate model must be evaluated against the complete collection before deployment. Production deep learning is not just model training; it is the feedback loop connecting models, data, people, evaluation, and the world in which the model operates.

## Why Deep Learning Had to Wait Until 2012

Backpropagation worked in 1986, and LeCun’s networks were reading zip codes and check numbers by the early 1990s. Yet neural networks remained a niche pursuit through most of the following two decades. The learning loop existed, but the models were difficult to optimize, large labeled datasets were scarce, and CPUs could not run sufficiently large experiments in a reasonable time. Deep learning did not take over because of one new algorithm. Several missing pieces arrived together.

In 2006, Geoffrey Hinton and Ruslan Salakhutdinov showed that deep networks could be coaxed into training by initializing their layers one at a time. The workaround helped revive interest in an approach many researchers had abandoned. But the decisive changes were about scale: enough data to constrain a large function, and hardware able to search its parameters.

Most of the training loop is **matrix multiplication**. Arrange a batch of inputs as one grid and a layer’s weights as another, and their product calculates every neuron’s weighted sum for every example. The cells can be computed largely independently. The backward pass reduces to more products of the same kind. A CPU is optimized to execute a small number of complicated instruction streams with low latency; this workload instead asks for enormous numbers of similar multiply and add operations at once.

The XOR network’s hidden layer is small enough to examine whole. On the left is the wiring from the earlier simulators; on the right, the same computation is represented as three grids: a mini-batch holding the four cases, the layer’s weights , and the weighted sums they produce. Click a cell of , a row of , a column of , or a neuron itself, and watch which parts of the other panel light up. Every cell of is one neuron’s weighted sum over one example; none of them reads any other, and that is the whole reason this workload suits the hardware we discuss in the next paragraph. Scale the same idea to a real layer, thousands of inputs by thousands of neurons against a batch of hundreds, and one such product is billions of multiply and adds that could all be performed in parallel. This is also the unit the field measures itself in, when a training run is quoted in floating-point operations, almost every one of them is a multiply and add inside a product like this.

Coincidentally, another industry had spent two decades accidentally building the perfect hardware for this workload. Rendering video game graphics means computing millions of pixels per frame, each by the same small program, none depending on another, the exact same shape of work as a layer of neurons.

The **GPU** (Graphics Processing Unit) evolved to serve it: thousands of simple cores executing one instruction across thousands of data elements simultaneously, maximum throughput in place of the CPU’s versatility. In 2007, NVIDIA released **CUDA**, a programming platform that let this machinery be aimed at arbitrary computation instead of pixels, on hardware any graduate student could buy in a consumer electronics store. (I’ll save the details of GPU architecture for my article on GPU Architecture.)

## how the GPU actually multiplies matrices

Thousands of cores are useless if they spend most of their time waiting for data to arrive. In matrix-heavy workloads, the true bottleneck is rarely the arithmetic calculations, it is **memory bandwidth** (how fast data moves from main memory to compute cores).

Fortunately, matrix multiplication naturally allows **data reuse**. Multiplying two matrices requires reading input numbers from memory, but performs roughly arithmetic operations, because every row of the first matrix is paired with every column of the second. That means every single number fetched from main memory can serve different calculations on-chip before being discarded.

The dark intersection is one dot product; each highlighted input value is reused across 6 output cells.

To exploit this, real GPU implementations never stream matrices straight through from main memory. Instead, they chop matrices into small **tiles** that fit inside fast, on-chip memory:

**Tile in Shared Memory:** A group of cores loads input tiles into fast, on-chip shared memory once.**Accumulate in Registers:** Each core keeps running totals of its output cells inside fast registers, computing entire blocks locally.**Write Back Once:** Cores write the finished block to main memory only when computation is complete.

The impact of this data-reuse strategy on performance is massive:

**Naive streaming**(re-reading main memory for every calculation):**~1%** of a GPU’s peak speed.**Tiling in shared memory**:**~10%** of peak speed.**Accumulating blocks in registers**:**~70%** of peak speed.**Hand-tuned vendor libraries**:**>90%** of peak speed.

Eventually, hardware design evolved to optimize matrix operations directly in silicon. Modern GPUs feature **tensor cores**—specialized hardware units that multiply matrix tiles in a single instruction—and rely on lower-precision math (16-bit or 8-bit) to pack more operations onto the same chip.

To overcome the memory bandwidth bottleneck, memory had to be redesigned too. AI accelerators moved to **HBM** (High Bandwidth Memory), stacking memory chips vertically right next to compute cores with thousands of direct connections to avoid round-trips to main memory. As deep learning grew in popularity, it pushed chip manufacturers to reshape their hardware around these exact workloads.

The second missing ingredient for scaling these learning algorithms was data. In 2009, Fei-Fei Li’s lab at Stanford released ImageNet: millions of labeled photographs across thousands of object categories, assembled through years of painstaking annotation work. The annual recognition competition built around it starting in 2010, spanning over a million training images across a thousand categories, gave the field a shared benchmark and an objective leaderboard.

GPU-trained convolutional networks had already begun winning niche contests. In 2011, Dan Cireșan and Jürgen Schmidhuber’s group took first place in traffic sign and handwriting benchmarks using neural networks trained on GPUs, proof that classic architectures, when powered by modern hardware, could outperform hand-crafted algorithms. What ImageNet offered was an undeniable, field-wide scoreboard where a victory would signal genuine progress across general computer vision.

In 2012, Ilya Sutskever convinced fellow Toronto graduate student Alex Krizhevsky to train a convolutional neural network on ImageNet, with Geoffrey Hinton serving as advisor.

Krizhevsky adapted custom CUDA code originally written for smaller datasets, extending it to run in parallel across two consumer gaming GPUs (NVIDIA GTX 580s) set up in his bedroom.

Over a year of constant tweaking and retraining, they optimized the parameters to create **AlexNet**. Their 2012 paper, [ “ImageNet Classification with Deep Convolutional Neural Networks”](https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html), would go on to become one of the most cited papers in computing history. Reflecting on the project years later, Hinton summarized it succinctly:

*“Ilya thought we should do it, Alex made it work, and I got the Nobel Prize.”*

The breakthrough ended a forty-year debate virtually overnight. AlexNet cut the top-5 error rate, the fraction of test images where the correct label appears nowhere among the model’s top five guesses—nearly in half, down to 15.3% (compared to 26.2% for the second-place hand-engineered system). The result was so decisive that within two years, virtually every major computer vision team had switched to deep networks. In 2025, Google partnered with the Computer History Museum to release the original 2012 AlexNet source code to the public.

AlexNet supplied the decisive public experiment: learned representations, backpropagation, abundant data, and parallel hardware now beat carefully engineered vision systems together. The argument shifted from whether neural networks could work to how far the same approach could scale.

## what scaling unlocked: AlphaGo, AlphaFold, and scientific recognition

The public saw another kind of learned program in 2016, when DeepMind’s AlphaGo, led by David Silver with DeepMind co-founder Demis Hassabis, defeated Lee Sedol, one of the greatest Go players in history. Unlike IBM’s earlier chess system Deep Blue, whose evaluation relied heavily on hand-crafted rules, AlphaGo used learned **policy** and **value** networks to choose promising moves and evaluate positions. It first learned from expert games and then improved through self-play.

In Game 2, AlphaGo played the celebrated **Move 37**, a high shoulder hit that contradicted centuries of conventional Go judgment. Commentators initially suspected a mistake before its strategic purpose became clear. In Game 4, Lee answered with his own remarkable Move 78, handing AlphaGo its only loss of the match.

Four years later, AlphaFold 2 achieved near-experimental accuracy on many protein-structure predictions in the CASP14 blind assessment. It displaced much of the hand-engineered pipeline with a function learned from known biological structures, and its released database made those predictions available to researchers at enormous scale.

The field’s institutional reversal was just as striking. Yann LeCun, Geoffrey Hinton, and Yoshua Bengio shared the 2018 Turing Award; John Hopfield and Hinton received the 2024 Nobel Prize in Physics for foundational work related to neural networks; and Demis Hassabis and John Jumper shared half of the 2024 Nobel Prize in Chemistry for AlphaFold.

## Removing the Sequential Bottleneck

Convolutional neural networks fit GPU hardware naturally: sliding filters process every position at once, so vision models scaled as fast as GPUs did. Text was a different story.

The standard sequence architectures, the **Recurrent Neural Network (RNN)** and its descendant, the **Long Short-Term Memory network (LSTM)**, process text word by word, the way a person reads: consume , update a hidden state to , carry that state forward to process , and repeat down the sequence, each state summarizing context so far.

Published in 1997 by Sepp Hochreiter and Jürgen Schmidhuber, the LSTM made long chains trainable by wrapping each step in **gates**, small learned sigmoid units that control what the carried state keeps, discards, and exposes. This created a direct highway for information to flow straight down the sequence without getting repeatedly multiplied down at every step, acting like a skip connection running across time steps instead of network depth, and solving the vanishing gradient problem Hochreiter had diagnosed six years earlier.

That carried state introduced a fundamental bottleneck: because is computed from , which is computed from , we couldn’t parallelize this computation. A 100-word sentence takes 100 dependent steps no matter how much hardware is available, meaning sequence models were bottlenecked by dependent steps, unable to take full advantage of the thousands of parallel compute cores modern GPUs provided.

Left: `h₃`

waits for `h₂`

, which waits for `h₁`

, one dependent step per word. Right: self-attention connects every word to every other word in a single parallel step.

The bridge out of this bottleneck began with **attention**. In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio showed in [ “Neural Machine Translation by Jointly Learning to Align and Translate”](https://arxiv.org/abs/1409.0473) that a translation model did not need to squeeze an entire source sentence into one fixed vector. When producing each output word, it could learn to assign different weights to the relevant input words and gather the information it needed at that moment. Attention was another learned weighted sum, but its weights depended on the current input rather than being permanently stored model parameters. It gave the network a form of content-addressable retrieval: look across the available context and pull forward what matters now.

The **transformer**, introduced in 2017 by Ashish Vaswani and seven Google colleagues in [ “Attention Is All You Need”](https://arxiv.org/abs/1706.03762), made this retrieval operation the architecture’s center and removed recurrence from the training path.

**Self-attention** lets every position gather information directly from every other position instead of carrying one state word by word. The positions in a training sequence can therefore be processed in parallel through matrix multiplications. The transformer still consists of weighted sums and non-linearities trained by the same four-step loop; it changed which relationships were easy to compute and learn.

## Self-Supervision: Training Without Labels

Nothing so far connects a digit classifier to a system that writes essays, code, or proofs. The bridge is surprisingly small. A language model is still a classifier: its input is a stretch of text, its possible categories are the tokens in a vocabulary, and its target is whichever token came next in the original text. Softmax turns the scores into a distribution and cross-entropy rewards probability assigned to the observed continuation, just as they do for image categories.

Nobody needs to label each example manually because ordinary text contains its own target, every prefix is followed by its actual continuation. This **self-supervision** removed the annotation bottleneck that had required years of human labor for ImageNet and allowed training datasets to grow by orders of magnitude.

Turning that classifier into a generator requires only a loop: choose a token from the predicted distribution, append it to the input, and predict again. A paragraph, proof, or program is produced one iteration at a time, with each output becoming part of the next input. The training objective sounds narrow, but reducing next-token error across varied writing rewards representations of syntax, facts, style, code, and recurring reasoning patterns, because all of them help predict what follows.

## How Models Receive Feedback

The thermometer, digit recognizer, AlphaGo, and language model learn through the same optimization loop, but their target answers come from different places. Most deep learning fits into three broad arrangements:

-
**Supervised learning:** another process supplies the desired answer. A human labels a photograph`cat`

; a laboratory instrument supplies the temperature; a database records a protein structure. The loss compares the prediction directly with that target. Supervision is precise, but producing enough trustworthy labels can be slow and expensive. -
**Self-supervised learning:** the data hides part of itself and asks the model to reconstruct or predict it. The next word labels a text prefix; surrounding pixels provide a target for a masked image patch; one view of an object can supervise another view. No person must annotate every example, so the model can learn from far larger collections. The task is chosen because solving it forces useful representations to emerge, even when that task is not the final product. -
**Reinforcement learning:** an agent takes actions, observes their consequences, and receives a**reward** rather than a correct answer for every step. The difficult part is credit across time: a move may appear harmless now but decide a game much later. AlphaGo’s self-play generated entire games, used the eventual outcome as evidence, and improved policies that made winning trajectories more likely. Reinforcement learning is suited to sequential decisions, but it can be unstable, data-hungry, and dangerously literal about exploiting an imperfect reward.

01

### Supervised

*→*target “cat”

*→*loss

02

### Self-supervised

*→*target “down”

*→*loss

03

### Reinforcement

*→*reward +1

*→*objective

These categories can be combined. AlphaGo began with supervised examples of expert moves and continued with reinforcement learning from self-play. Modern foundation models commonly begin with self-supervised pretraining and then undergo supervised or reinforcement-based post-training. What changes between the stages is not backpropagation, but where the target or reward comes from and therefore what behavior the loss encourages.

## Learning to Generate

A classifier learns a boundary between categories. A **generative model** learns enough of the structure of its training distribution to create new samples resembling it. The next-token loop is one method: generate a complex object one piece at a time, always conditioning on the pieces already produced. Images, audio, and video created another branch of the story.

In 2014, Ian Goodfellow and collaborators introduced the **Generative Adversarial Network (GAN)** in [ “Generative Adversarial Nets”](https://arxiv.org/abs/1406.2661). A generator tried to create convincing samples while a discriminator tried to distinguish generated samples from real data. Because both were neural networks, the discriminator’s error could flow backward into the generator, teaching it which changes would make its output harder to detect. GANs produced striking images, but training two networks locked in competition was notoriously unstable: one could overpower the other, and a generator could learn only a narrow selection of outputs.

**Diffusion models** supplied a more stable mental model. Begin with a real example, add random noise in many small steps until its structure disappears, and train a network to predict how to reverse that corruption. To generate, start from random noise and repeatedly denoise it. Each step turns an unstructured sample into one that lies slightly closer to patterns the network learned from data.

```
Autoregressive generation: predict the next piece, append it, repeat
Diffusion generation:      predict the noise, remove some, repeat
```

Move the noise slider to compare a corrupted sample with the noise a trained model would estimate and the slightly cleaner result of subtracting it. **Generate from noise** animates the repeated reverse process.

The modern diffusion lineage includes score-based modeling by Yang Song and Stefano Ermon and the influential 2020 [ Denoising Diffusion Probabilistic Model](https://arxiv.org/abs/2006.11239) by Jonathan Ho, Ajay Jain, and Pieter Abbeel. Later systems moved some of this process into compressed latent spaces and extended the same principle across images, speech, music, and video. The architecture may still be a convolutional network or transformer; what makes it diffusion is the corruption and denoising training objective.

Both branches reveal the same underlying principle. A loss does not need to describe the final creative act directly. It only needs to create millions of small prediction problems whose solution forces the network to capture the structure required for generation.

## One Model Becomes the Starting Point for Many

AlexNet represented an earlier pattern: assemble a labeled dataset for one task, train one model, and deploy it for that task. Modern deep learning increasingly begins with **pretraining** one model on a broad objective and reusing the representations it learns.

Suppose a vision network has already learned edges, textures, shapes, objects, and scenes. A new medical-image classifier need not rediscover every low-level feature from random weights. Engineers can keep much of the pretrained network, replace its final output layer, and continue training on the smaller specialized dataset. This is **transfer learning**. **Fine-tuning** updates some or all of the pretrained weights; parameter-efficient methods instead train small added modules while leaving most weights frozen.

The reusable internal coordinates are often called **embeddings**. An embedding converts an item, a word, image, sound, molecule, or user, into a vector whose geometry reflects distinctions useful to the training objective. Items used in similar ways or paired across the data tend to occupy nearby regions. Searching, clustering, recommendation, and cross-modal retrieval can then operate on learned geometry rather than raw pixels or characters.

The simulator below offers a simplified view. Real embeddings may have thousands of dimensions and cannot be displayed directly on a two-dimensional screen. Select a relationship or drag a point: meaning is represented by relative positions and directions, not by a dictionary entry stored beside each word.

OpenAI’s [ CLIP](https://arxiv.org/abs/2103.00020) made this shift especially visible in 2021. It learned from 400 million image-text pairs by bringing the embedding of each image closer to its matching caption and farther from incorrect captions. After training, written descriptions could name new visual categories without adding a dedicated output neuron or retraining on the usual labeled examples. Language had become an interface to a reusable visual representation.

A broadly pretrained model that can be adapted to many downstream uses is now called a **foundation model**, a term [formalized by Stanford researchers in 2021](https://crfm.stanford.edu/report.html). Adaptation can happen by fine-tuning weights, adding a task-specific head or adapter, retrieving external information, or when the model is large enough, placing instructions and demonstrations in its input without changing any weights at all. This **in-context learning** changes the computation performed for the current request while the underlying model remains frozen.

This is the dominant present-day pattern: broad data and a general objective produce one pretrained foundation model, then task context or adaptation turns it into many specialized systems.

The advantage is leverage: the enormous cost of learning general representations is paid once and reused. The risk is inheritance: downstream systems also inherit the foundation model’s blind spots, biases, copyrighted or sensitive traces, security weaknesses, and poorly understood internal behavior. One model becoming the base of many products concentrates both capability and failure.

## Scaling the Search

Once transformers could efficiently absorb self-supervised text, researchers found that adding parameters, data, and compute often reduced loss in smooth, predictable ways. Empirical **scaling laws** estimate those relationships and help allocate a training budget between model size and data. The important change was cultural as much as mathematical, progress increasingly came from running a general learning method at a scale that hand-engineered systems could not match.

“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”

Richard Sutton, “The Bitter Lesson” (2019)

The lesson is bitter because general methods that exploit computation repeatedly overtake systems built from more satisfying hand-crafted solutions. The frontier consequently became industrial as well as algorithmic: curating enormous datasets, coordinating accelerators, moving data quickly enough to feed them, and keeping a training run stable for months. Scaling transformers on text produced the large language models of the current era, which I will examine in more detail in a future article.

## Where the Frontier Stands Today

The transformer and scaling laws did not finish the story. They produced a powerful general substrate, and current research asks how to make that substrate reason more deliberately, perceive and act across modalities, learn with less data and energy, and become understandable enough to trust. Five directions capture much of the frontier.

### Models Are Becoming Multimodal

Text, images, audio, video, and actions are no longer treated as unrelated problems with permanently separate models. Each modality can be converted into sequences or embeddings, letting a shared network connect a spoken instruction to a video, a diagram to an explanation, or a camera observation to a robot action. The important advance is not simply accepting more file types. It is learning a shared representation in which information acquired through one modality can help interpret or generate another.

Multimodality also exposes a limitation of learning from text alone. Language records a compressed description of the world, but omits much of its geometry, timing, causality, and physical texture. Video, interaction, and sensor data provide different evidence. How to align these sources without allowing the easiest modality to dominate the others remains an open problem.

### Computation Is Moving Into Inference

The first scaling era concentrated computation before deployment, train a larger model on more data, then run one forward pass per prediction. A growing line of work instead lets a frozen model spend more effort on difficult inputs. It can produce several candidates, search through intermediate steps, use a learned verifier, revise an answer, or call external tools. Experiments on [scaling test-time computation](https://arxiv.org/abs/2408.03314) show that allocating this effort well can sometimes beat using a much larger model for the same problem.

This changes the economics and mental model of inference. An answer is no longer necessarily one sample from one forward pass; it can be the result of a small search conducted by a learned model. The frontier is deciding when extra computation helps, how to verify intermediate work, and how to prevent a longer search from merely producing a more elaborate mistake.

### Models Are Learning Environments and Actions

A **world model** learns how a state may change after an action. Instead of only predicting the next token in a document, it may predict the next video frame, physical state, or sensory observation. Such models can generate interactive simulations, help an agent plan before acting, or provide synthetic experience for robotics. Systems such as DeepMind’s [Genie 3](https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/) illustrate the direction: learned generative environments becoming substrates in which agents can be trained and evaluated.

The hard part is grounding. A visually plausible future is not necessarily physically correct, and a robot cannot tolerate the kinds of small invented details that are harmless in an image. Embodied systems must connect representation learning to control, memory, uncertainty, real-time constraints, and safe recovery when reality differs from prediction.

### Scale Is Becoming Selective

Larger dense models apply all their parameters to every input, even when most are irrelevant. **Mixture of Experts** (MoE) models instead learn a router that activates only a subset of specialized parameter blocks for each token or example. Modern sparse systems such as William Fedus, Barret Zoph, and Noam Shazeer’s [Switch Transformer](https://arxiv.org/abs/2101.03961) demonstrated how a model could add far more parameters without using all of them for every token. Other techniques compress a capable teacher into a smaller student, lower numerical precision, cache reusable computations, or fine-tune small adapters instead of the whole network.

Data is becoming selective too. Once the easiest public data has been consumed, progress depends more on deduplication, provenance, filtering, synthetic examples, simulation, active learning, and data engines that target known weaknesses. Continual learning, adding new knowledge without erasing old capabilities, remains difficult because gradient updates that help the new distribution can damage representations learned from the old one. The next scaling era is therefore as much about choosing computation and evidence well as increasing their raw quantity.

### We Still Cannot Reliably Explain the Program We Found

Deep learning discovers programs we could not write, but those programs are consequently difficult to inspect. Researchers can identify some learned features and trace portions of individual computations. Work on [mapping features inside production language models](https://www.anthropic.com/research/mapping-mind-language-model), for example, finds distributed representations corresponding to concrete and abstract concepts and shows that manipulating some of them changes behavior.

But recognizing features is not the same as explaining a whole answer, guaranteeing what a model will do in a new situation, or proving that an apparently safe policy will remain safe under pressure. Interpretability, adversarial robustness, uncertainty calibration, evaluation, privacy, alignment with human intent, and control of autonomous behavior remain active research problems.

The [Stanford AI Index](https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance) documents the tension defining the present state: measured capabilities continue to improve rapidly, while evaluation, reliability, cost, transparency, and responsible deployment lag behind. Deep learning is simultaneously an established engineering method and a scientific object whose most capable examples are not yet understood in the way traditional programs are.

## What to Keep in Mind

At its core, deep learning searches for a mathematical function that maps inputs to useful outputs. Training adjusts the function’s parameters until its behavior scores well against an objective; inference runs the function that search discovered. The result is not a collection of readable rules, but a program distributed across learned numbers: capable where training rewarded useful regularities and unreliable where the objective or evidence left its behavior unconstrained.

Five consequences follow from this idea:

**Data defines the available evidence.** A model cannot learn a distinction its examples never reveal, and it will absorb accidental patterns that consistently help prediction.**The objective defines success.** Gradient descent does not pursue truth or human intent directly. It pursues whatever behavior lowers the loss.**Architecture shapes the search.** Convolutions make repeated spatial patterns cheap, attention makes relationships between positions direct, and every architectural choice makes some programs easier to discover than others.**Performance on unseen cases is the real test.** A low training loss proves only that fitting occurred. Held-out data, counterexamples, and changed conditions reveal whether the intended pattern generalizes.**Reuse concentrates consequences.** A foundation model can spread useful representations across thousands of applications, but its hidden failures and assumptions travel with them.

Traditional software engineering requires an expert to understand a procedure well enough to write it down as step-by-step instructions. Deep learning asks us to provide experience and define what success looks like, letting search discover the procedure instead. We trade readable code for behaviors we could not have written ourselves and inherit the responsibility of understanding what that search actually rewarded.