# AI Fundamentals: Understanding Activation Functions (Part 1) > Source: > Published: 2026-08-02 20:31:01+00:00 Stacking a hundred layers in a neural network without non-linear activation functions causes the entire architecture to suffer from **linear collapse**. Mathematically, every linear layer performs an *affine transformation*: a combination of matrix multiplication and vector addition, y = Wx + **b**. Because the composition of any number of affine transformations is itself just another affine transformation, a network with ten, a hundred, or a thousand linear layers simplifies directly into a single matrix operation: output = Wₑ · x + **bₑ** Breaking the above equation down further: Layer 1: y₁ = W₁x + **b**₁ Layer 2: y₂ = W₂y₁ + **b**₂ Layer 3: y₃ = W₃y₂ + **b**₃ Plugging each layer into the next: y₃ = W₃(W₂(W₁x + **b**₁) + **b**₂) + **b**₃ Multiplying them: y₃ = (W₃W₂W₁)x + (W₃W₂**b**₁ + W₃** b**₂ + **b**₃) Instead of carrying those nested matrices around, group them into two variables: Wₑ = W₃W₂W₁ (the effective overall weight matrix) and, **b**ₑ = W₃W₂** b**₁ + W₃** b**₂ + **b**₃ (the effective overall bias vector). The entire 3-layer network collapses right back into that same single-layer formula: output = Wₑ · x + **bₑ** Picture looking down at a map with a single small island surrounded entirely by ocean, then being handed a ruler and asked to draw one straight line that puts every bit of land on one side and every bit of water on the other. There’s no way to do it: any straight line drawn across that map cuts through both the island and the ocean around it. What’s needed instead is a nonlinear boundary that can wrap around the island and separate it from the surrounding ocean. That’s the intuition behind what a neural network learns. Rather than being limited to simple straight-line separations, neural networks learn transformations that reshape data into representations where complex decision surfaces become possible. So, when we talk about a network “separating datasets,” the real meaning is that it learns a decision function that divides the input space into regions: everything on one side belongs to class A, and everything on the other belongs to class B. Whether that boundary is a straight line, a curve, a circle, or a far more complex shape depends entirely on how the data is arranged. An activation function is a small non-linear operation applied after each layer’s linear step: squashing, clipping, or reshaping the output before it moves on. Instead of z = W₂(W₁x + **b**₁) + **b**₂, the result becomes something like z = W₂·f(W₁x + **b**₁) + **b**₂, where f is a non-linear function like a sigmoid, tanh, ReLU, etc. The activation function doesn’t need to be complicated to do its job. Even ReLU, which is max(0, x), a function that just clips negative values to zero, is enough to stop the network from collapsing into a single linear transformation. That single f breaks the algebra: there’s no matrix M and vector c such that f(W₁x + **b**₁) = Wx + **b**, for every x. Stack enough of these non-linear steps together, and the network stops being restricted to straight-line thinking; it can carve out circles, spirals, and shapes. That’s the whole *purpose* of an activation function, at the most fundamental level: it’s the thing standing between “a network that can only draw straight lines” and “a network that can wrap a boundary around almost any shape thrown at it.” Each neuron by itself contributes one tiny bend and a network is thousands of them, each bending things in a slightly different spot. Stack enough of them together, and the network can approximate curves and boundaries that no single neuron, or even a hundred of them, could pull off alone. Take an example: *“Time flies like an arrow; fruit flies like a banana.”* Read the first half and “flies” is a verb: time is moving, fast, like an arrow. Read the second half and “flies” is a noun: fruit flies are a kind of insect that seems to enjoy bananas. Same word, wildly different job, and the only thing signaling which is which is the surrounding context. A model has to somehow pull those two uses of “flies” apart into different regions of its internal representation, even though at the input level they’re the identical token. This is where depth and non-linearity earn their keep together. Because [each layer starts from a different random point](https://pub.towardsai.net/ai-fundamentals-what-is-a-neuron-62e76a6f3500), each one ends up drawing its bent boundary through the data in a slightly different place. As training proceeds, this quiet divergence gets shaped into something closer to specialization. The example above is an over-simplification: real models don’t cleanly assign “this layer = nouns, that layer = verbs” in a tidy labeled way, but a loose intuition is: earlier layers could pick up on more local, surface-level patterns (word order, part of speech, etc.), while deeper layers integrate more surrounding context and start representing something closer to *meaning*, which sense of “flies” is active, what “it” refers to, that sort of thing. It’s specifically the bending, layer after layer, that gives the network enough room to gradually tease “time flies” and “fruit flies” apart into different corners of its representation space, instead of being stuck treating “flies” as one fixed thing no matter what’s around it. The UAT states that a feedforward neural network with a non-linear activation function and a sufficiently large hidden layer can, in principle, approximate any continuous function on a bounded domain to any desired degree of accuracy. One intuitive way to understand this capability is by imagining how networks combine many simple nonlinear components to create increasingly complex shapes and behaviors. These components can be thought of as localized building blocks, or “towers,” that each contribute a small part of the final function. By combining enough of these building blocks, a network can gradually approximate an arbitrary continuous function as closely as desired. This same principle explains why neural networks can represent everything from simple decision boundaries to the complex internal representations used in tasks such as language understanding. To understand the whole system, look at how raw neurons turn into local pixels, and how those pixels draw a curve. **1. The core problem with single neurons** A single ReLU neuron is global. Once it switches on, it never turns back off, so its influence isn’t confined to any one region; it keeps affecting every point further along the x-axis too. That means there’s no way to use a single raw neuron to nudge the curve at one specific location without that same nudge bleeding into every location past it, since the neuron has no way to turn back off and contain the effect. **2. The solution: combining neurons into localized “bumps”** Each neuron sets a threshold where it fires. Combining multiple neurons creates an active window, allowing the network to control local regions without affecting the entire domain. Example, Each step represents the active window created by that pair of neurons, a range from x = 2 to x = 4 where they're allowed to influence the final answer. **3. Reading the graph** On a coordinate plane, each tower maps directly to network parameters: The Universal Approximation Theorem is an existence proof, not a construction manual: it guarantees that a network capable of approximating almost any function exists somewhere in the space of possible networks, but it doesn’t say how many neurons that takes, whether gradient descent can actually find the right weights, or how well the result generalizes. Representing a function and learning it from data are two different problems, and the UAT only answers the first one. Everything modern networks actually rely on, depth, attention, residual connections, optimizers, huge datasets, careful initialization, sits entirely outside what the theorem promises. It’s not an algorithm a network runs during training, nor is it a recipe for building one. Instead, it’s the mathematical justification for why introducing non-linearity fundamentally changes what a neural network is capable of representing in the first place. [AI Fundamentals: Understanding Activation Functions (Part 1)](https://pub.towardsai.net/ai-fundamentals-understanding-activation-functions-part-1-fde15d65c735) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.