Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star git-lrc on GitHub to help more developers discover the project. Do give it a try and share your feedback for improving the product.
In the previous article, we derived formulas for updating the output layer weights w3, w4, and bias b3. Now, we will understand how to calculate the gradients for the hidden layer parameters: w1, b1, w2, and b2.
To find the gradients of the parameters in the hidden layer, we need to trace how changing these values affects the final prediction and the error (SSR).
Let's recall the structure of our neural network:
For the top neuron: x1 = input * w1 + b1
y1 = f(x1) = log(1 + e^x1) *(using the softplus function)*
For the bottom neuron:
x2 = input * w2 + b2
y2 = f(x2) = log(1 + e^x2) (using the softplus function) Finally, the prediction:
Predicted = y1 * w3 + y2 * w4 + b3
And the prediction error:
SSR = Σ (observed − predicted)² Since w1, b1, w2, and b2 are not directly connected to the output prediction, we must use the chain rule to backpropagate the error from the output layer back to the hidden layer.
Let's calculate the gradient for the top neuron's weight w1 first.
A change in w1 affects x1, which affects the output y1, which affects the predicted value, which finally affects the SSR.
So, by the chain rule:
dSSR/dw1 = dSSR/d(predicted) * d(predicted)/dy1 * dy1/dx1 * dx1/dw1 Let's calculate each of these values:
As we saw in the previous articles, this is the derivative of SSR with respect to the predicted value:
dSSR/d(predicted) = -2 * (Observed - Predicted) Since Predicted = y1 * w3 + y2 * w4 + b3, and all other terms are treated as constants w.r.t y1:
d(predicted)/dy1 = w3 Since y1 = log(1 + e^x1), the derivative of the softplus function is the logistic sigmoid function:
dy1/dx1 = e^x1 / (1 + e^x1) Since x1 = input * w1 + b1, differentiating w.r.t w1 gives:
dx1/dw1 = input
Multiplying these parts together, we get:
dSSR/dw1 = -2 * (Observed - Predicted) * w3 * (e^x1 / (1 + e^x1)) * input Similarly, for the top neuron's bias b1:
dSSR/db1 = dSSR/d(predicted) * d(predicted)/dy1 * dy1/dx1 * dx1/db1 The only term that changes here is the last one:
dx1/db1 = 1 (since x1 = input * w1 + b1, derivative w.r.t b1 is 1) So:
dSSR/db1 = -2 * (Observed - Predicted) * w3 * (e^x1 / (1 + e^x1)) * 1 Following the same logic, we can find the gradients for the bottom neuron's parameters:
dSSR/dw2 = dSSR/d(predicted) * d(predicted)/dy2 * dy2/dx2 * dx2/dw2
dSSR/dw2 = -2 * (Observed - Predicted) * w4 * (e^x2 / (1 + e^x2)) * input
dSSR/db2 = dSSR/d(predicted) * d(predicted)/dy2 * dy2/dx2 * dx2/db2
dSSR/db2 = -2 * (Observed - Predicted) * w4 * (e^x2 / (1 + e^x2)) * 1
Once we calculate all these derivatives (dSSR/dw1, dSSR/db1, dSSR/dw2, dSSR/db2), we can update the hidden layer weights and biases using gradient descent:
Step size w1 = derivation w1 * Learning rate
New w1 = old w1 - Step size w1 Step size b1 = derivation b1 * Learning rate
New b1 = old b1 - Step size b1 Step size w2 = derivation w2 * Learning rate
New w2 = old w2 - Step size w2 Step size b2 = derivation b2 * Learning rate
New b2 = old b2 - Step size b2 By doing this repeatedly, the model minimizes the error and converges to the optimal values for all weights and biases.
We have successfully derived the formulas to calculate the gradients for w1, b1, w2, and b2. Combined with the output layer derivations, we now have the math for the entire neural network's backpropagation!
In the next article, we will see how to implement this in code.
Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.