{"slug": "i-finally-understood-why-neural-networks-need-activation-functions", "title": "I Finally Understood Why Neural Networks Need Activation Functions", "summary": "A developer implemented Sigmoid, Tanh, ReLU, Leaky ReLU, and Softmax from scratch in Python and realized that the derivative of an activation function is what actually teaches the neural network how to learn. The activation function determines the output during the forward pass, while its derivative computes gradients during backpropagation to update weights. The developer noted that ReLU's derivative (0 for negative inputs, 1 for positive) explains why it trains deep networks faster than Sigmoid, which suffers from vanishing gradients.", "body_md": "When I started learning Deep Learning, I thought activation functions were just mathematical equations we had to memorize.\n\nToday, while implementing **Sigmoid, Tanh, ReLU, Leaky ReLU, and Softmax from scratch in Python**, I realized something much more interesting.\n\nThe activation function itself is only **half of the story**.\n\nThe **derivative** is what actually teaches the neural network how to learn.\n\nI generated **100 equally spaced values between -10 and 10**:\n\n```\nz = np.linspace(-10, 10, 100)\n```\n\nThen I implemented:\n\nFor the first four, I plotted **both the activation function and its derivative**.\n\nInitially, I wondered:\n\nWhy am I plotting the derivative? Isn't the activation function enough?\n\nAfter learning about **backpropagation**, the answer became clear.\n\nThe activation function answers:\n\n\"What output should this neuron produce?\"\n\nFor example, Sigmoid compresses any input into a value between **0 and 1**.\n\nReLU outputs **0** for negative inputs and passes positive inputs unchanged.\n\nThis is what happens during the **forward pass**.\n\n```\nInput\n   ↓\nWeighted Sum\n   ↓\nActivation Function\n   ↓\nPrediction\n```\n\nThe derivative answers a completely different question:\n\n\"How much should this neuron change to reduce the error?\"\n\nDuring the **backward pass**, the neural network computes gradients.\n\nThose gradients come directly from the **derivatives of the activation functions**.\n\nWithout derivatives, the model wouldn't know how to update its weights.\n\n```\nPrediction\n     ↓\nCalculate Error\n     ↓\nCompute Derivatives\n     ↓\nUpdate Weights\n```\n\nThat was the moment when the plots finally made sense to me.\n\nBeautiful S-shaped curve.\n\nBut its derivative almost becomes **zero** for very large positive or negative inputs.\n\nThat means learning slows down because the gradients almost disappear.\n\nThis is called the **vanishing gradient problem**.\n\nAt first glance, it looked almost identical to Sigmoid.\n\nBut I noticed something important:\n\nIts output is centered around **zero**.\n\nThat small difference helps optimization because the gradients are more balanced around the origin.\n\nThe simplest graph was also the most surprising.\n\n```\nf(x)=max(0,x)\n```\n\nIts derivative is\n\n```\n0   (negative inputs)\n\n1   (positive inputs)\n```\n\nThis explains why ReLU trains deep neural networks much faster than Sigmoid.\n\nReLU has one weakness.\n\nNegative neurons completely stop learning because the derivative becomes zero.\n\nLeaky ReLU fixes that with a tiny negative slope.\n\nInstead of\n\n```\nGradient = 0\n```\n\nit becomes\n\n```\nGradient = 0.01\n```\n\nA small change mathematically—but an important one during training.\n\nSoftmax was different from the others.\n\nIt doesn't transform a single value.\n\nIt transforms an **entire vector into probabilities**.\n\nExample:\n\n```\nInput\n\n[2,4,1]\n\n↓\n\nOutput\n\n[0.114\n 0.844\n 0.042]\n```\n\nEvery probability depends on every other input.\n\nThat explains why we usually don't visualize Softmax as a simple curve.\n\nToday I stopped thinking of activation functions as just formulas.\n\nNow I see them as two complementary ideas:\n\nThe graph explains the prediction.\n\nThe derivative explains the learning.\n\nMy next goal is to implement:\n\nI want to understand the mathematics before relying on deep learning frameworks.\n\nWhen you first learned neural networks, what concept changed your understanding the most—activation functions, backpropagation, or gradient descent?", "url": "https://wpnews.pro/news/i-finally-understood-why-neural-networks-need-activation-functions", "canonical_source": "https://dev.to/karki_vasagnavi/i-finally-understood-why-neural-networks-need-activation-functions-2cm6", "published_at": "2026-07-21 15:06:14+00:00", "updated_at": "2026-07-21 15:22:59.408908+00:00", "lang": "en", "topics": ["neural-networks", "machine-learning"], "entities": ["Python", "Sigmoid", "Tanh", "ReLU", "Leaky ReLU", "Softmax"], "alternates": {"html": "https://wpnews.pro/news/i-finally-understood-why-neural-networks-need-activation-functions", "markdown": "https://wpnews.pro/news/i-finally-understood-why-neural-networks-need-activation-functions.md", "text": "https://wpnews.pro/news/i-finally-understood-why-neural-networks-need-activation-functions.txt", "jsonld": "https://wpnews.pro/news/i-finally-understood-why-neural-networks-need-activation-functions.jsonld"}}