In the earliest days, activation functions were literal binary gates. If an input crossed a specific threshold, the neuron output 1; otherwise, it output 0. While intuitive, it had a fundamental limitation: its derivative made it unsuitable for conventional gradient-based optimization and backpropagation.
To enable smooth, continuous learning, researchers transitioned to S-shaped bounding functions. **Sigmoid **compresses any real number into the open range from 0 to 1, while **Tanh **stretches that range from -1 to 1.
At first glance, this seemed ideal. A value between 0 and 1 echoes the idea of a biological neuron either firing or staying quiet. However, this design choice hid a flaw for deep networks: the vanishing gradient problem.
The sigmoid curve has a steep slope near its center but approaches flat saturation at large positive and negative inputs.
Here’s why that flatness is a problem. Training a network means calculating an error at the end, then working backward through each layer to figure out how much each weight contributed to that error. To do this, the network multiplies gradients together layer by layer, using the chain rule:
If even a few of those gradients are close to zero, which happens whenever a neuron sits in the flat part of the Sigmoid or Tanh curve, the multiplication drags the whole product toward zero.
Multiply small gradients across many layers, and the error signal can become negligibly small. The earliest layers stop receiving any meaningful update, effectively frozen in place while the rest of the network continues to learn around them.
You can observe this collapse directly in Python:
import numpy as np# Used during forward pass# Turns a weighted sum into activated output(value between 0 and 1)def sigmoid(x): return 1 / (1 + np.exp(-x))# Used during backward pass # Measures how sensitive the output is to change in xdef sigmoid_grad(x): s = sigmoid(x) return s * (1 - s)# Simulate a backward pass through a tiny 5-layer network.# Each layer just multiplies by a weight, then applies Sigmoidnp.random.seed(0) num_layers = 5# Apply a random weight per layerweights = np.random.randn(num_layers) * 1.5 # A single starting inputx = 3.0 # Forward pass: save the pre-activation value (the weighted sum only)# This is used by sigmoid_grad to compute each layer's local gradient.pre_activations = []a = xfor w in weights: z = a * w #the weighted sum entering this layer pre_activations.append(z) a = sigmoid(z) # This layer's output becomes the next layer's input# Backward pass: start with a gradient of 1.0 at the output# Multiply by each layer's local gradient# (and its weight) as the signal travels back toward layer 1.grad = 1.0for i in reversed(range(num_layers)): grad *= sigmoid_grad(pre_activations[i]) * weights[i] print(f"Gradient after stepping back through layer {i + 1}: {grad:.8f}")
Most of the curve’s real estate is flat with only a narrow sliver near zero actually having meaningful slope. And since the gradient is literally the correction signal backprop uses to update a weight, a near-flat gradient means a near-nothing correction, no matter how wrong the network’s output actually was. The weight doesn’t get meaningfully nudged toward “more correct”; it just stays parked almost exactly where it already was. Stack several layers where this keeps happening, and most of the network is stuck getting these practically-nothing corrections, batch after batch, which is exactly why Sigmoid and Tanh do fine in shallow networks, but can make optimization difficult on deep networks.
Rectified Linear Unit (ReLU) solves this problem by abandoning squashing entirely. Its logic is brutally simple: if an input is positive, let it pass untouched; if it is negative, zero it out.
ReLU(x) = max(0, x)
For positive inputs, ReLU’s derivative is 1, so it avoids the gradient attenuation caused by sigmoid-like saturation. However, gradients can still vanish or explode because of the network’s weights and architecture.
Because ReLU’s gradient is strictly 0 for any negative input, a neuron that consistently receives negative values can become permanently inactive. It outputs zero, produces zero gradient, and never updates again, a phenomenon known as the dying ReLU problem.
To combat this, researchers developed variations like Leaky ReLU,** **which leaves a tiny, non-zero slope for negative inputs and Parametric ReLU (PReLU), which allows the network to learn that negative slope dynamically during training.
def relu(x): return np.maximum(0, x) def relu_grad(x): return np.where(x > 0, 1.0, 0.0) for x in [-2, 0.1, 5]: print(f"x={x}: ReLU Grad = {relu_grad(x)}")
To fix the sharp corners and dead zones, modern architectures increasingly use smooth nonlinearities such as GELU (Gaussian Error Linear Unit) and SiLU (Sigmoid Linear Unit, also known as Swish), particularly in Transformer-based models.
Instead of making a hard binary choice at zero, these functions use smooth transitions. A slightly negative input might be scaled down rather than entirely wiped out, allowing a struggling neuron to maintain a faint training signal and crawl its way back.
def silu(x): return x * (1 / (1 + np.exp(-x)))for x in [-2, -1, 0, 1, 2]: print(f"x={x}, SiLU = {silu(x):.4f}")
Both GELU and SiLU provide smoother gradients and can support effective training trajectories, allowing top-tier model architectures to combine these with gating mechanisms (such as SwiGLU), multiplying an activated path by a raw content path to maximize expressive power.
Traditional activation functions apply a fixed nonlinear transformation to each activation, whereas gated architectures use learned projections to dynamically modulate one representation with another.
Instead of running a single calculation, a gated layer takes the exact same input and splits it into two independent processing paths using separate sets of weights:
Multiplying the gate and content together elementwise allows the network to modulate information contextually rather than relying on a fixed activation curve. The elementwise multiplication allows one projected representation to modulate another.
Below is a simplified implementation sample:
import numpy as npclass SwiGLULayer: def __init__(self, in_features, hidden_features): # 1. Gate projection matrix self.gate_weights = np.random.randn(in_features, hidden_features) * np.sqrt(2.0 / in_features) # 2. Content projection matrix self.content_weights = np.random.randn(in_features, hidden_features) * np.sqrt(2.0 / in_features) # 3. Down-projection matrix self.down_weights = np.random.randn(hidden_features, in_features) * np.sqrt(2.0 / hidden_features) # Cache for backpropagation self.input_cache = None self.gate_pre_activation = None self.gate_final_output = None self.content_output = None self.elementwise_product = None def forward(self, input_tensor): self.input_cache = input_tensor # 1. Compute Gate path: linear projection followed by SiLU activation self.gate_pre_activation = np.dot(input_tensor, self.gate_weights) sigmoid_gate = 1 / (1 + np.exp(-self.gate_pre_activation)) self.gate_final_output = self.gate_pre_activation * sigmoid_gate # 2. Compute Content path: linear projection of the content payload self.content_output = np.dot(input_tensor, self.content_weights) # 3. Element-wise multiplication: the gate dynamically scales the content self.elementwise_product = self.gate_final_output * self.content_output # 4. Down-projection: maps back to the original model dimension output_tensor = np.dot(self.elementwise_product, self.down_weights) return output_tensor def backward(self, upstream_gradient): # Gradient through down-projection # d_ : stands for derivative, indicating how much the weights need to be modified at each layer moving backwards d_down_weights = np.dot(self.elementwise_product.T, upstream_gradient) d_elementwise = np.dot(upstream_gradient, self.down_weights.T) # Gradient split across the element-wise multiplication product d_gate_out = d_elementwise * self.content_output d_content_out = d_elementwise * self.gate_final_output # Gradient through Content path weights d_content_weights = np.dot(self.input_cache.T, d_content_out) # Gradient through SiLU activation on the Gate path sig = 1 / (1 + np.exp(-self.gate_pre_activation)) d_gate_pre = d_gate_out * sig * (1.0 + self.gate_pre_activation * (1.0 - sig)) # Gradient through Gate path weights d_gate_weights = np.dot(self.input_cache.T, d_gate_pre) # Gradient flowing back to the preceding layer's input d_input = np.dot(d_gate_pre, self.gate_weights.T) + np.dot(d_content_out, self.content_weights.T) return d_input, d_gate_weights, d_content_weights, d_down_weights# Example executionnp.random.seed(42)batch_size, in_dim, hidden_dim = 2, 4, 8# A mini batch of input vectors (e.g., token representations)x_input = np.random.randn(batch_size, in_dim)layer = SwiGLULayer(in_features=in_dim, hidden_features=hidden_dim)output = layer.forward(x_input)print("Input shape:", x_input.shape)print("SwiGLU output shape:", output.shape)print("Sample output values:\n", output)
in_features — size of the incoming feature dimension, or how many numbers are coming into the layer for a single sample.
hidden_features — is the expanded internal size of the layer. This layer usually takes an input(x), blows it up into a much larger hidden space to let the network think and mix features, and then shrinks it back down.
np.sqrt(2.0 / in_features) — is the scaling factor used in this example as a simple initialization strategy. It shrinks or stretches the random initial numbers just enough to keep the variance of the signals stable as they flow forward.
linear_projections — imagine input tokens packed into a tight, low-dimensional space. Different features might end up tangled together or compressed. By blowing it up into a massive hidden space, the network has room to separate those intertwined features, let individual neurons specialize, and mix information across different dimensions cleanly.
Because gated architectures have demonstrated strong optimization and modeling performance in modern language models, the computational cost can be worthwhile.
Activation function design has evolved from rigid binary steps into an intentional engineering discipline focused on precise gradient control, dynamic sparsity, and adaptive architectures. As AI continues to scale, the search for smarter ways to route and transform information remains an active frontier.
AI Fundamentals: Understanding Activation Functions (Part 2) was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.