{"slug": "ai-fundamentals-understanding-activation-functions-part-2", "title": "AI Fundamentals: Understanding Activation Functions (Part 2)", "summary": "A new technical explainer on activation functions highlights the vanishing gradient problem that plagues Sigmoid and Tanh in deep networks, and introduces ReLU as a solution. The article includes Python code demonstrating how gradients collapse through a 5-layer network, showing the gradient shrinking to near zero. ReLU's derivative of 1 for positive inputs avoids this attenuation, though the article notes potential issues with negative inputs.", "body_md": "In the earliest days, activation functions were literal binary gates. If an input crossed a specific threshold, the neuron output 1; otherwise, it output 0. While intuitive, it had a fundamental limitation: its derivative made it unsuitable for conventional gradient-based optimization and backpropagation.\n\nTo enable smooth, continuous learning, researchers transitioned to S-shaped bounding functions. **Sigmoid **compresses any real number into the open range from 0 to 1, while **Tanh **stretches that range from -1 to 1.\n\nAt first glance, this seemed ideal. A value between 0 and 1 echoes the idea of a biological neuron either firing or staying quiet. However, this design choice hid a flaw for deep networks: **the vanishing gradient problem**.\n\nThe sigmoid curve has a steep slope near its center but approaches flat saturation at large positive and negative inputs.\n\nHere’s why that flatness is a problem. Training a network means calculating an error at the end, then working backward through each layer to figure out how much each weight contributed to that error. To do this, the network multiplies gradients together layer by layer, using the chain rule:\n\nIf even a few of those gradients are close to zero, which happens whenever a neuron sits in the flat part of the Sigmoid or Tanh curve, the multiplication drags the whole product toward zero.\n\nMultiply small gradients across many layers, and the error signal can become negligibly small. The earliest layers stop receiving any meaningful update, effectively frozen in place while the rest of the network continues to learn around them.\n\nYou can observe this collapse directly in Python:\n\n``` python\nimport numpy as np# Used during forward pass# Turns a weighted sum into activated output(value between 0 and 1)def sigmoid(x):    return 1 / (1 + np.exp(-x))# Used during backward pass # Measures how sensitive the output is to change in xdef sigmoid_grad(x):    s = sigmoid(x)    return s * (1 - s)# Simulate a backward pass through a tiny 5-layer network.# Each layer just multiplies by a weight, then applies Sigmoidnp.random.seed(0) num_layers = 5# Apply a random weight per layerweights = np.random.randn(num_layers) * 1.5 # A single starting inputx = 3.0                                        # Forward pass: save the pre-activation value (the weighted sum only)# This is used by sigmoid_grad to compute each layer's local gradient.pre_activations = []a = xfor w in weights:    z = a * w #the weighted sum entering this layer    pre_activations.append(z)    a = sigmoid(z) # This layer's output becomes the next layer's input# Backward pass: start with a gradient of 1.0 at the output# Multiply by each layer's local gradient# (and its weight) as the signal travels back toward layer 1.grad = 1.0for i in reversed(range(num_layers)):    grad *= sigmoid_grad(pre_activations[i]) * weights[i]    print(f\"Gradient after stepping back through layer {i + 1}: {grad:.8f}\")\n```\n\nMost of the curve’s real estate is flat with only a narrow sliver near zero actually having meaningful slope. And since the gradient is literally the correction signal backprop uses to update a weight, a near-flat gradient means a near-nothing correction, no matter how wrong the network’s output actually was. The weight doesn’t get meaningfully nudged toward “more correct”; it just stays parked almost exactly where it already was. Stack several layers where this keeps happening, and most of the network is stuck getting these practically-nothing corrections, batch after batch, which is exactly why Sigmoid and Tanh do fine in shallow networks, but can make optimization difficult on deep networks.\n\nRectified Linear Unit (**ReLU**) solves this problem by abandoning squashing entirely. Its logic is brutally simple: if an input is positive, let it pass untouched; if it is negative, zero it out.\n\nReLU(x) = max(0, x)\n\nFor positive inputs, ReLU’s derivative is 1, so it avoids the gradient attenuation caused by sigmoid-like saturation. However, gradients can still vanish or explode because of the network’s weights and architecture.\n\nBecause ReLU’s gradient is strictly 0 for any negative input, a neuron that consistently receives negative values can become permanently inactive. It outputs zero, produces zero gradient, and never updates again, a phenomenon known as the **dying ReLU problem**.\n\nTo combat this, researchers developed variations like **Leaky ReLU**,** **which leaves a tiny, non-zero slope for negative inputs and **Parametric ReLU (PReLU)**, which allows the network to learn that negative slope dynamically during training.\n\n``` python\ndef relu(x):         return np.maximum(0, x)  def relu_grad(x):         return np.where(x > 0, 1.0, 0.0)  for x in [-2, 0.1, 5]:         print(f\"x={x}: ReLU Grad = {relu_grad(x)}\")\n```\n\nTo fix the sharp corners and dead zones, modern architectures increasingly use smooth nonlinearities such as **GELU** (Gaussian Error Linear Unit) and **SiLU** (Sigmoid Linear Unit, also known as Swish), particularly in Transformer-based models.\n\nInstead of making a hard binary choice at zero, these functions use smooth transitions. A slightly negative input might be scaled down rather than entirely wiped out, allowing a struggling neuron to maintain a faint training signal and crawl its way back.\n\n``` python\ndef silu(x):    return x * (1 / (1 + np.exp(-x)))for x in [-2, -1, 0, 1, 2]:    print(f\"x={x}, SiLU = {silu(x):.4f}\")\n```\n\nBoth GELU and SiLU provide smoother gradients and can support effective training trajectories, allowing top-tier model architectures to combine these with gating mechanisms (such as **SwiGLU**), multiplying an activated path by a raw content path to maximize expressive power.\n\nTraditional activation functions apply a fixed nonlinear transformation to each activation, whereas gated architectures use learned projections to dynamically modulate one representation with another.\n\nInstead of running a single calculation, a gated layer takes the exact same input and splits it into two independent processing paths using separate sets of weights:\n\nMultiplying the gate and content together elementwise allows the network to modulate information contextually rather than relying on a fixed activation curve. The elementwise multiplication allows one projected representation to **modulate** another.\n\nBelow is a simplified implementation sample:\n\n``` python\nimport numpy as npclass SwiGLULayer:    def __init__(self, in_features, hidden_features):        # 1. Gate projection matrix        self.gate_weights = np.random.randn(in_features, hidden_features) * np.sqrt(2.0 / in_features)        # 2. Content projection matrix        self.content_weights = np.random.randn(in_features, hidden_features) * np.sqrt(2.0 / in_features)        # 3. Down-projection matrix        self.down_weights = np.random.randn(hidden_features, in_features) * np.sqrt(2.0 / hidden_features)        # Cache for backpropagation        self.input_cache = None        self.gate_pre_activation = None        self.gate_final_output = None        self.content_output = None        self.elementwise_product = None    def forward(self, input_tensor):        self.input_cache = input_tensor               # 1. Compute Gate path: linear projection followed by SiLU activation        self.gate_pre_activation = np.dot(input_tensor, self.gate_weights)        sigmoid_gate = 1 / (1 + np.exp(-self.gate_pre_activation))        self.gate_final_output = self.gate_pre_activation * sigmoid_gate             # 2. Compute Content path: linear projection of the content payload        self.content_output = np.dot(input_tensor, self.content_weights)        # 3. Element-wise multiplication: the gate dynamically scales the content        self.elementwise_product = self.gate_final_output * self.content_output        # 4. Down-projection: maps back to the original model dimension        output_tensor = np.dot(self.elementwise_product, self.down_weights)        return output_tensor    def backward(self, upstream_gradient):        # Gradient through down-projection        # d_ : stands for derivative, indicating how much the weights need to be modified at each layer moving backwards        d_down_weights = np.dot(self.elementwise_product.T, upstream_gradient)        d_elementwise = np.dot(upstream_gradient, self.down_weights.T)                # Gradient split across the element-wise multiplication product        d_gate_out = d_elementwise * self.content_output        d_content_out = d_elementwise * self.gate_final_output        # Gradient through Content path weights        d_content_weights = np.dot(self.input_cache.T, d_content_out)        # Gradient through SiLU activation on the Gate path        sig = 1 / (1 + np.exp(-self.gate_pre_activation))        d_gate_pre = d_gate_out * sig * (1.0 + self.gate_pre_activation * (1.0 - sig))        # Gradient through Gate path weights        d_gate_weights = np.dot(self.input_cache.T, d_gate_pre)        # Gradient flowing back to the preceding layer's input        d_input = np.dot(d_gate_pre, self.gate_weights.T) + np.dot(d_content_out, self.content_weights.T)        return d_input, d_gate_weights, d_content_weights, d_down_weights# Example executionnp.random.seed(42)batch_size, in_dim, hidden_dim = 2, 4, 8# A mini batch of input vectors (e.g., token representations)x_input = np.random.randn(batch_size, in_dim)layer = SwiGLULayer(in_features=in_dim, hidden_features=hidden_dim)output = layer.forward(x_input)print(\"Input shape:\", x_input.shape)print(\"SwiGLU output shape:\", output.shape)print(\"Sample output values:\\n\", output)\n```\n\nin_features — **size of the incoming feature dimension**, or how many numbers are coming into the layer for a single sample.\n\nhidden_features — is the **expanded internal size** of the layer. This layer usually takes an input(x), blows it up into a much larger hidden space to let the network think and mix features, and then shrinks it back down.\n\nnp.sqrt(2.0 / in_features) — is the **scaling factor** used in this example as a simple initialization strategy. It shrinks or stretches the random initial numbers just enough to keep the variance of the signals stable as they flow forward.\n\nlinear_projections — imagine input tokens packed into a tight, low-dimensional space. Different features might end up tangled together or compressed. By blowing it up into a massive hidden space, the network has room to separate those intertwined features, let individual neurons specialize, and mix information across different dimensions cleanly.\n\nBecause gated architectures have demonstrated strong optimization and modeling performance in modern language models, the computational cost can be worthwhile.\n\nActivation function design has evolved from rigid binary steps into an intentional engineering discipline focused on precise gradient control, dynamic sparsity, and adaptive architectures. As AI continues to scale, the search for smarter ways to route and transform information remains an active frontier.\n\n[AI Fundamentals: Understanding Activation Functions (Part 2)](https://pub.towardsai.net/ai-fundamentals-understanding-activation-functions-part-2-ac15aa63b15b) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/ai-fundamentals-understanding-activation-functions-part-2", "canonical_source": "https://pub.towardsai.net/ai-fundamentals-understanding-activation-functions-part-2-ac15aa63b15b?source=rss----98111c9905da---4", "published_at": "2026-08-17 13:31:01+00:00", "updated_at": "2026-08-17 13:42:25.386466+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks"], "entities": ["Sigmoid", "Tanh", "ReLU", "Python"], "alternates": {"html": "https://wpnews.pro/news/ai-fundamentals-understanding-activation-functions-part-2", "markdown": "https://wpnews.pro/news/ai-fundamentals-understanding-activation-functions-part-2.md", "text": "https://wpnews.pro/news/ai-fundamentals-understanding-activation-functions-part-2.txt", "jsonld": "https://wpnews.pro/news/ai-fundamentals-understanding-activation-functions-part-2.jsonld"}}