Andrej Karpathy's Neural Networks: Zero to Hero — 1) Intro to Neural Networks and Backpropagation

Andrej Karpathy released a series of lecture videos and open-source code titled "Neural Networks: Zero to Hero," beginning with a deep dive into backpropagation. In the first lecture, Karpathy built a small project called *micrograd* to demonstrate how neural networks function under the hood, arguing that backpropagation is the essential mechanism for training networks while everything else is primarily for efficiency. The project and accompanying code are available on GitHub.

Andrej Karpathy uploaded several lecture videos on YouTube and the accompanying code on GitHub. I think they are excellent lectures, even better than many paid online courses. Here's the link: Neural Networks: Zero to Hero https://github.com/karpathy/nn-zero-to-hero . So, I will summarize them and try to get meaningful insights from them. I'm going to cover all of them lecture by lecture I hope... The first lecture is about backpropagation in neural networks. In the video, Karpathy said that backpropagation is what you need to train neural networks, and everything else is mainly for efficiency. That is why he explained and demonstrated backpropagation in the very first video. I totally agree with him. Training neural networks and LLMs is essentially about reducing loss. And fundamentally, all methods for reducing loss are related to backpropagation, directly or indirectly. Karpathy built a small project called micrograd . You can see the code here https://github.com/karpathy/micrograd . This is made up of just a few simple lines of code, but it shows us how neural networks are built under the hood. In the video, he demonstrated how to build Micrograd and how it works step by step. We all learned differentiation and derivatives , right? What do differentiation and derivatives actually mean? Differentiation and derivatives tell us how much f x f x f x changes when xxx changes. That is, they show the effect of a variable and the slope or gradient of a function. Then, if the derivative is 0, that point may be a local maximum or minimum of the function—not always, but it can be. Actually, Karpathy didn't explain differentiation and derivatives in detail in the video. However, I think this is one of the most important aspects for understanding neural networks. So I'm going to explain this in more detail. This concept is fundamental to linear regression as well. What is the core idea of linear regression? The goal is to minimize the residual sum of squares RSS . This is what linear regression is all about: finding the optimal β\betaβ values. Then, how to find them? This is where the derivative comes in. The RSS formula is a kind of quadratic function, so when it comes to quadratic functions, the minimum point is the point at which the derivative becomes 0. Therefore, if we differentiate the equation and find where the derivative is zero, we can find the best β1\beta 1β1 and β0\beta 0β0 . To find the optimal β0\beta 0β0 and β1\beta 1β1 , we take the partial derivatives of RSS with respect to each parameter and set them equal to zero. I will demonstrate how to derive them. First, differentiate the RSS with respect to β0\beta 0β0 : Then, differentiate the RSS with respect to β1\beta 1β1 : At the minimum point, both partial derivatives are zero: This gives us the normal equations: Solving these equations gives the optimal values: Here, xˉ\bar{x}xˉ is the mean of the input values, and yˉ\bar{y}yˉ is the mean of the target values. Neural networks are also built from these kinds of linear expressions, usually combined with nonlinear activation functions. However, the way we calculate the parameters is totally different because neural networks are much more complicated and have so many parameters. So it is almost impossible to find the parameters in this way. Finding your way in pitch darkness https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flyxf4v0f9t5q9ov6o4a8.png Assume that while hiking in the mountains, we get lost and trying to get down the mountain. But it is night, so we are in pitch darkness. We can only see a few inches around us. In this case, how can we get down the mountain? The answer is simple: by following the slope downward. At least, if we can see the slope around us, we can tell which way leads downward. This is how we find the optimal point when the equation is so complex that we are not able to solve for the optimum analytically. python def f x : return 3 x 2 - 4 x + 5 f 3.0 xs = np.arange -5, 5, 0.25 ys = f xs plt.plot xs, ys Visualization of the quadratic function https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvpebqpadyzrfsmex5o05.png This is an example from Karpathy's code. The function is f x =3x2−4x+5f x = 3x^2 - 4x + 5f x =3x2−4x+5 . Its derivative is df x dx=6x−4\frac{d f x }{d x} = 6x - 4dxdf x =6x−4 . If we solve the equation 6x−4=06x - 4 = 06x−4=0 , the derivative is 0 at x=2/3x = 2/3x=2/3 . Then, if xxx is at some other point, how can we move xxx to find the minimum point of f x f x f x ? The answer is simple. If the derivative value at a certain point is greater than 0, we have to decrease xxx ; if it is less than 0, we have to increase xxx . Therefore, if we consistently subtract λ 6x−4 \lambda 6x - 4 λ 6x−4 from xxx , with a proper learning rate λ\lambdaλ , f x f x f x will converge to its minimum. This finite-difference approach is useful for building intuition and for gradient checking. h = 0.000001 x = 2/3 f x + h - f x /h The output is 2.999378523327323e-06, which is almost zero. It is not perfectly exact because floating-point numbers have limited precision, and because this is a finite-difference approximation, but it is close enough for this simple demonstration. Similarly, when you have an expression with several variables, you can get the slope with respect to a specific variable in this way. h = 0.0001 inputs a = 2.0 b = -3.0 c = 10.0 d1 = a b + c c += h d2 = a b + c print 'd1', d1 print 'd2', d2 print 'slope', d2 - d1 /h Output: d1 4.0 d2 4.0001 slope 0.9999999999976694 This is also Karpathy's code. This example shows how ddd changes when ccc changes from 10.0. The gradient is 1.0, of course, since the derivative ddc ab+c \frac{d}{dc} ab + c dcd ab+c is 1. Karpathy demonstrates a hands-on example of how to calculate the gradients. Let's see one of his examples. When fff is the final output, we should calculate the gradients with respect to all intermediate values. Let's do this one by one. The gradient of fff with respect to itself is 1 because ddff=1\frac{d}{df}f = 1dfdf=1 . Easy, right? The important thing is that addition passes the gradient through. For example, if x=a+bx = a + bx=a+b and ddxf x =k\frac{d}{dx}f x = kdxdf x =k , the gradients of aaa and bbb are also kkk . For subtraction, the subtracted term receives the negative of the upstream gradient. And when it comes to multiplication, the upstream gradient is multiplied by the other variable. If x=a×bx = a \times bx=a×b , the gradient of aaa is k×bk \times bk×b because the local derivative with respect to aaa is bbb . Therefore, the gradient of eee is ddef\frac{d}{de}fdedf . fff is d×ed \times ed×e , so the gradient eee is d=−6.0d=-6.0d=−6.0 . On the other hand, the gradient of ddd is 1.0. Finally, the gradient of aaa is the contribution from eee plus the contribution from ddd . So it is −6+3=−3-6 + 3 = -3−6+3=−3 . The gradient of bbb is also the contribution from eee plus the contribution from ddd , and that is −6−2=−8-6 - 2 = -8−6−2=−8 . The computation graph for the final output looks like this: What if we want to minimize fff by tuning the value of aaa ? By subtracting the gradient, −3λ-3\lambda−3λ , fff will get smaller since the function with respect to aaa is an upward-opening quadratic function. If λ\lambdaλ is 0.1, we subtract -0.3 from aaa . Then aaa becomes -1.7. As a result, fff becomes -6.63, which is smaller. Now, let's apply this algorithm to a neural network. I organized another example from Karpathy in the image above. This is a very simple neural network architecture that he made. Actually, this is not the whole story yet. What we want to minimize is the loss function. So, if ooo is y^\hat{y}y^ , gradient descent should minimize ∑i=1n yi−oi 2\sum {i=1}^{n} \left y i - o i \right ^2∑i=1n yi−oi 2 In this way, Karpathy shows hands-on code that runs gradient descent on a simple MLP. Here's the MLP training loop Karpathy builds in the Micrograd lecture: python class Neuron: def init self, nin : self.w = Value random.uniform -1,1 for in range nin self.b = Value random.uniform -1,1 def call self, x : w x + b act = sum wi xi for wi, xi in zip self.w, x , self.b out = act.tanh return out def parameters self : return self.w + self.b class Layer: def init self, nin, nout : self.neurons = Neuron nin for in range nout def call self, x : outs = n x for n in self.neurons return outs 0 if len outs == 1 else outs def parameters self : return p for neuron in self.neurons for p in neuron.parameters class MLP: def init self, nin, nouts : sz = nin + nouts self.layers = Layer sz i , sz i+1 for i in range len nouts def call self, x : for layer in self.layers: x = layer x return x def parameters self : return p for layer in self.layers for p in layer.parameters x = 2.0, 3.0, -1.0 n = MLP 3, 4, 4, 1 n x xs = 2.0, 3.0, -1.0 , 3.0, -1.0, 0.5 , 0.5, 1.0, 1.0 , 1.0, 1.0, -1.0 , ys = 1.0, -1.0, -1.0, 1.0 desired targets for k in range 20 : forward pass ypred = n x for x in xs loss = sum yout - ygt 2 for ygt, yout in zip ys, ypred backward pass for p in n.parameters : p.grad = 0.0 loss.backward update for p in n.parameters : p.data += -0.1 p.grad print k, loss.data With his Micrograd code, you can see how the gradient descent algorithm works step by step. You can also calculate the gradient of each variable on your own. I strongly recommend doing these hands-on examples. After watching the video, I was able to clearly understand how gradient descent works, why we should use zero out gradients, why ReLU function is the most efficient, and so on. This is definitely worth your time. I have shown Karpathy's demonstrations of gradient descent. As he said, this is the core concept for training neural networks. The rest is just for efficiency. Reducing loss using gradients: this is what makes neural network training possible and, ultimately, helped usher in the AI era.