When I started learning Deep Learning, I thought activation functions were just mathematical equations we had to memorize.
Today, while implementing Sigmoid, Tanh, ReLU, Leaky ReLU, and Softmax from scratch in Python, I realized something much more interesting.
The activation function itself is only half of the story.
The derivative is what actually teaches the neural network how to learn.
I generated 100 equally spaced values between -10 and 10:
z = np.linspace(-10, 10, 100)
Then I implemented:
For the first four, I plotted both the activation function and its derivative.
Initially, I wondered:
Why am I plotting the derivative? Isn't the activation function enough?
After learning about backpropagation, the answer became clear.
The activation function answers:
"What output should this neuron produce?"
For example, Sigmoid compresses any input into a value between 0 and 1.
ReLU outputs 0 for negative inputs and passes positive inputs unchanged.
This is what happens during the forward pass.
Input
↓
Weighted Sum
↓
Activation Function
↓
Prediction
The derivative answers a completely different question:
"How much should this neuron change to reduce the error?"
During the backward pass, the neural network computes gradients.
Those gradients come directly from the derivatives of the activation functions.
Without derivatives, the model wouldn't know how to update its weights.
Prediction
↓
Calculate Error
↓
Compute Derivatives
↓
Update Weights
That was the moment when the plots finally made sense to me.
Beautiful S-shaped curve.
But its derivative almost becomes zero for very large positive or negative inputs.
That means learning slows down because the gradients almost disappear.
This is called the vanishing gradient problem.
At first glance, it looked almost identical to Sigmoid.
But I noticed something important:
Its output is centered around zero.
That small difference helps optimization because the gradients are more balanced around the origin.
The simplest graph was also the most surprising.
f(x)=max(0,x)
Its derivative is
0 (negative inputs)
1 (positive inputs)
This explains why ReLU trains deep neural networks much faster than Sigmoid.
ReLU has one weakness.
Negative neurons completely stop learning because the derivative becomes zero.
Leaky ReLU fixes that with a tiny negative slope.
Instead of
Gradient = 0
it becomes
Gradient = 0.01
A small change mathematically—but an important one during training.
Softmax was different from the others.
It doesn't transform a single value.
It transforms an entire vector into probabilities.
Example:
Input
[2,4,1]
↓
Output
[0.114
0.844
0.042]
Every probability depends on every other input.
That explains why we usually don't visualize Softmax as a simple curve.
Today I stopped thinking of activation functions as just formulas.
Now I see them as two complementary ideas:
The graph explains the prediction.
The derivative explains the learning.
My next goal is to implement:
I want to understand the mathematics before relying on deep learning frameworks.
When you first learned neural networks, what concept changed your understanding the most—activation functions, backpropagation, or gradient descent?