AI Fundamentals: Understanding Activation Functions (Part 2)

A new technical explainer on activation functions highlights the vanishing gradient problem that plagues Sigmoid and Tanh in deep networks, and introduces ReLU as a solution. The article includes Python code demonstrating how gradients collapse through a 5-layer network, showing the gradient shrinking to near zero. ReLU's derivative of 1 for positive inputs avoids this attenuation, though the article notes potential issues with negative inputs.

In the earliest days, activation functions were literal binary gates. If an input crossed a specific threshold, the neuron output 1; otherwise, it output 0. While intuitive, it had a fundamental limitation: its derivative made it unsuitable for conventional gradient-based optimization and backpropagation. To enable smooth, continuous learning, researchers transitioned to S-shaped bounding functions. Sigmoid compresses any real number into the open range from 0 to 1, while Tanh stretches that range from -1 to 1. At first glance, this seemed ideal. A value between 0 and 1 echoes the idea of a biological neuron either firing or staying quiet. However, this design choice hid a flaw for deep networks: the vanishing gradient problem . The sigmoid curve has a steep slope near its center but approaches flat saturation at large positive and negative inputs. Here’s why that flatness is a problem. Training a network means calculating an error at the end, then working backward through each layer to figure out how much each weight contributed to that error. To do this, the network multiplies gradients together layer by layer, using the chain rule: If even a few of those gradients are close to zero, which happens whenever a neuron sits in the flat part of the Sigmoid or Tanh curve, the multiplication drags the whole product toward zero. Multiply small gradients across many layers, and the error signal can become negligibly small. The earliest layers stop receiving any meaningful update, effectively frozen in place while the rest of the network continues to learn around them. You can observe this collapse directly in Python: python import numpy as np Used during forward pass Turns a weighted sum into activated output value between 0 and 1 def sigmoid x : return 1 / 1 + np.exp -x Used during backward pass Measures how sensitive the output is to change in xdef sigmoid grad x : s = sigmoid x return s 1 - s Simulate a backward pass through a tiny 5-layer network. Each layer just multiplies by a weight, then applies Sigmoidnp.random.seed 0 num layers = 5 Apply a random weight per layerweights = np.random.randn num layers 1.5 A single starting inputx = 3.0 Forward pass: save the pre-activation value the weighted sum only This is used by sigmoid grad to compute each layer's local gradient.pre activations = a = xfor w in weights: z = a w the weighted sum entering this layer pre activations.append z a = sigmoid z This layer's output becomes the next layer's input Backward pass: start with a gradient of 1.0 at the output Multiply by each layer's local gradient and its weight as the signal travels back toward layer 1.grad = 1.0for i in reversed range num layers : grad = sigmoid grad pre activations i weights i print f"Gradient after stepping back through layer {i + 1}: {grad:.8f}" Most of the curve’s real estate is flat with only a narrow sliver near zero actually having meaningful slope. And since the gradient is literally the correction signal backprop uses to update a weight, a near-flat gradient means a near-nothing correction, no matter how wrong the network’s output actually was. The weight doesn’t get meaningfully nudged toward “more correct”; it just stays parked almost exactly where it already was. Stack several layers where this keeps happening, and most of the network is stuck getting these practically-nothing corrections, batch after batch, which is exactly why Sigmoid and Tanh do fine in shallow networks, but can make optimization difficult on deep networks. Rectified Linear Unit ReLU solves this problem by abandoning squashing entirely. Its logic is brutally simple: if an input is positive, let it pass untouched; if it is negative, zero it out. ReLU x = max 0, x For positive inputs, ReLU’s derivative is 1, so it avoids the gradient attenuation caused by sigmoid-like saturation. However, gradients can still vanish or explode because of the network’s weights and architecture. Because ReLU’s gradient is strictly 0 for any negative input, a neuron that consistently receives negative values can become permanently inactive. It outputs zero, produces zero gradient, and never updates again, a phenomenon known as the dying ReLU problem . To combat this, researchers developed variations like Leaky ReLU , which leaves a tiny, non-zero slope for negative inputs and Parametric ReLU PReLU , which allows the network to learn that negative slope dynamically during training. python def relu x : return np.maximum 0, x def relu grad x : return np.where x 0, 1.0, 0.0 for x in -2, 0.1, 5 : print f"x={x}: ReLU Grad = {relu grad x }" To fix the sharp corners and dead zones, modern architectures increasingly use smooth nonlinearities such as GELU Gaussian Error Linear Unit and SiLU Sigmoid Linear Unit, also known as Swish , particularly in Transformer-based models. Instead of making a hard binary choice at zero, these functions use smooth transitions. A slightly negative input might be scaled down rather than entirely wiped out, allowing a struggling neuron to maintain a faint training signal and crawl its way back. python def silu x : return x 1 / 1 + np.exp -x for x in -2, -1, 0, 1, 2 : print f"x={x}, SiLU = {silu x :.4f}" Both GELU and SiLU provide smoother gradients and can support effective training trajectories, allowing top-tier model architectures to combine these with gating mechanisms such as SwiGLU , multiplying an activated path by a raw content path to maximize expressive power. Traditional activation functions apply a fixed nonlinear transformation to each activation, whereas gated architectures use learned projections to dynamically modulate one representation with another. Instead of running a single calculation, a gated layer takes the exact same input and splits it into two independent processing paths using separate sets of weights: Multiplying the gate and content together elementwise allows the network to modulate information contextually rather than relying on a fixed activation curve. The elementwise multiplication allows one projected representation to modulate another. Below is a simplified implementation sample: python import numpy as npclass SwiGLULayer: def init self, in features, hidden features : 1. Gate projection matrix self.gate weights = np.random.randn in features, hidden features np.sqrt 2.0 / in features 2. Content projection matrix self.content weights = np.random.randn in features, hidden features np.sqrt 2.0 / in features 3. Down-projection matrix self.down weights = np.random.randn hidden features, in features np.sqrt 2.0 / hidden features Cache for backpropagation self.input cache = None self.gate pre activation = None self.gate final output = None self.content output = None self.elementwise product = None def forward self, input tensor : self.input cache = input tensor 1. Compute Gate path: linear projection followed by SiLU activation self.gate pre activation = np.dot input tensor, self.gate weights sigmoid gate = 1 / 1 + np.exp -self.gate pre activation self.gate final output = self.gate pre activation sigmoid gate 2. Compute Content path: linear projection of the content payload self.content output = np.dot input tensor, self.content weights 3. Element-wise multiplication: the gate dynamically scales the content self.elementwise product = self.gate final output self.content output 4. Down-projection: maps back to the original model dimension output tensor = np.dot self.elementwise product, self.down weights return output tensor def backward self, upstream gradient : Gradient through down-projection d : stands for derivative, indicating how much the weights need to be modified at each layer moving backwards d down weights = np.dot self.elementwise product.T, upstream gradient d elementwise = np.dot upstream gradient, self.down weights.T Gradient split across the element-wise multiplication product d gate out = d elementwise self.content output d content out = d elementwise self.gate final output Gradient through Content path weights d content weights = np.dot self.input cache.T, d content out Gradient through SiLU activation on the Gate path sig = 1 / 1 + np.exp -self.gate pre activation d gate pre = d gate out sig 1.0 + self.gate pre activation 1.0 - sig Gradient through Gate path weights d gate weights = np.dot self.input cache.T, d gate pre Gradient flowing back to the preceding layer's input d input = np.dot d gate pre, self.gate weights.T + np.dot d content out, self.content weights.T return d input, d gate weights, d content weights, d down weights Example executionnp.random.seed 42 batch size, in dim, hidden dim = 2, 4, 8 A mini batch of input vectors e.g., token representations x input = np.random.randn batch size, in dim layer = SwiGLULayer in features=in dim, hidden features=hidden dim output = layer.forward x input print "Input shape:", x input.shape print "SwiGLU output shape:", output.shape print "Sample output values:\n", output in features — size of the incoming feature dimension , or how many numbers are coming into the layer for a single sample. hidden features — is the expanded internal size of the layer. This layer usually takes an input x , blows it up into a much larger hidden space to let the network think and mix features, and then shrinks it back down. np.sqrt 2.0 / in features — is the scaling factor used in this example as a simple initialization strategy. It shrinks or stretches the random initial numbers just enough to keep the variance of the signals stable as they flow forward. linear projections — imagine input tokens packed into a tight, low-dimensional space. Different features might end up tangled together or compressed. By blowing it up into a massive hidden space, the network has room to separate those intertwined features, let individual neurons specialize, and mix information across different dimensions cleanly. Because gated architectures have demonstrated strong optimization and modeling performance in modern language models, the computational cost can be worthwhile. Activation function design has evolved from rigid binary steps into an intentional engineering discipline focused on precise gradient control, dynamic sparsity, and adaptive architectures. As AI continues to scale, the search for smarter ways to route and transform information remains an active frontier. AI Fundamentals: Understanding Activation Functions Part 2 https://pub.towardsai.net/ai-fundamentals-understanding-activation-functions-part-2-ac15aa63b15b was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.