Foundation of the of Deep Learning

Andrej Karpathy's Autograd, an automatic differentiation engine, forms the foundation of deep learning by calculating derivatives to adjust neural network parameters during training. The engine enables backpropagation, which computes gradients to minimize loss, and relies on components such as weights, biases, activation functions, and loss functions to learn from data. This mechanism underpins models like GPT, Claude, and Gemini.

I was always curious about algorithms running behind models like GPT, Claude, and Gemini, etc. So I was researching what is the actual algorithm/engine or foundation that makes these models train and predict things. As I was searching, I stumbled upon Andrej Karpathy’s “Autograd ” and what Foundation of Deep Learning - I was always curious about the algorithms behind models like GPT, Claude, and Gemini, etc.So http://etc.so/ I was researching what actual algorithm or foundation is used that makes these models train themselves and predict things.As http://things.as/ I was searching, I stumbled upon Andrej Karpathy’s “Autograd ”. In this blog, I will try to cover about each and every thing I possibly can, so if you don’t know much about coding, you would also be able to understand if you are just curious enough. The most basic working of Autograd is - It is an automatic differentiation engine that actually calculates the derivative or slope of the mathematical operation in your code. In non mathmatical term — it is an engine that calculates the exact adjustments needed to help the network learn and make predictions So, what a neural network does is: it takes input data , parameters weights and biases , and it performs some mathematical operations like multiplications and additions , stores all the calculations inside nodes, and makes a prediction; that is what we call a forward pass. After it makes its first prediction, it calculates the Loss how far the prediction is from the actual answer , and we call this a loss function. Now, to make the model predict the correct one by minimizing the gap between the prediction and the actual answer i.e loss, it triggers backpropagation, reading its hidden history map nodes in reverse and finding gradients, which means showing how much each input weight contributed to the total error. Prerequisite Information - What are weights?Weight controls the strength of the connection between two neurons. It tell/decides how much influence does the input have on the final output. Working - Every piece of the incoming data is multiplied by its corresponding weights. The math = Output = input weight Example — House Price prediction, the number of bedrooms and the color of the front door are the input. The network will assign a heavy positive weight to bedroom because they matter a lot, and a neat-zero weight to the door color as it is irrelevant What are biases?Bias is an extra number that added to the sum of your weighted inputs. It allows you to shift your entire calculation up or down, completely independent of the inputs. Working - It act like a intercept © in the classic line equation y = mx + c. Even if all the inputs are zero, the biases ensures the neuron can still output a value. What is an activation function?Activation Function decided whether a neuron should “fire” pass information forward or not. Working - It takes the combined math of your inputs, weights, and biases input weight + bias and squashes it into a specific range like 0 to 1, or -1 to 1 without this, ai is just a giant calculator adding and multiplying numbers. It can only draw straight lines. It could never understand a curved pattern. Example - In house price prediction, a house cannot have a negative price. An activation function like ReLU will automatically turn any negative calculation into a clean zero, keeping the data realistic. What is Loss function? Loss Function is the network’s report card. It measures the error by calculating exaclty how far off the network’s prediction was from the actual true answer. Working - It takes the final output of your network that it predicted and compares it to the actual target /answer. Example - If the network predicts a house costs $300k, but it actually costs $400k, the loss function calculates that $100k gap and converts it into a single “error score” for Autograd to look at. What is gradient or derivative?Gradient in simple terms means rate of change. it measures the change happened at every node of the mathematical operation, denoting exactly how much that specific node is responsible for the network’s final error Working- It will twl you the slope of the loss function. If the gradient is +ve increasing the weight will increase the error. If it is -ve, increasing the weight will decrease the error. Example- Think of being blindfolded on a mountain the mountain is the error . The gradient tells you which direction is “downhill” so you can safely walk toward the bottom zero error . Actual flow - input → input integrating with paramters weights and biases going inside a neural network → output the prediction → find the loss → twike the numbers to minimize the error → repeat till prediction == target Understanding the code - class Value: python def init self, data, children= , op='', label = '' : self.data = data we use this in order to track the steps of the computation derivate a value node only now its current number, not how it go there self. prev give node a memory self. prev = set children self. op = op self.grad = 0.0 self. backward = lambda: None self.label = label python def repr self : return f"Value data = {self.data} " python def neg self : return self -1 python def sub self, other : return self + -other python def add self, other : other = other if isinstance other, Value else Value other out = Value self.data + other.data, children= self, other , op='+' python def backward : self.grad += 1.0 out.grad other.grad += 1.0 out.grad out. backward = backward return out python def mul self, other : other = other if isinstance other, Value else Value other out = Value self.data other.data, children = self, other , op=' ' python def backward : self.grad += other.data out.grad other.grad += self.data out.grad out. backward = backward return out python def rmul self, other : return self other python def truediv self, other : return self other -1 python def radd self, other : other + self return self + other python def tanh self : x = self.data t = math.exp 2 x - 1 / math.exp 2 x + 1 out = Value t, children = self, , op='tanh' python def backward : self.grad += 1 - t 2 out.grad out. backward = backward return out python def exp self : x = self.data out = Value math.exp x , self, , 'exp' python def backward : self.grad += out.data out.grad out. backward = backward return out python def pow self, other : assert isinstance other, int, float out = Value self.data other, self, , f" {other}" python def backward : self.grad += other self.data other -1 out.grad out. backward = backward return out python def backward self : topo = visited = set python def build topo v : if v not in visited: visited.add v for child in v. prev: build topo child topo.append v build topo self self.grad = 1 for node in reversed topo : node. backward So this is our Value object, the automatic differentiation engine. It is the core mathematical Backbone needed to train a Neural Network. Starting with - Methods / Functions for Mathematical operations — creating methods We write methods like add and mul to trick Python into using math symbols on custom objects. It includes type checking isinstance so mixing regular numbers like a + 2 doesn't cause a crash. It also recycles existing code: subtraction sub is re-engineered as adding a negative number, and division truediv is re-engineered as multiplying by a negative power. Building a Computational Graph Forward Pass inputs x1,x2x1 = Value 2.0, label='x1' x2 = Value 0.0, label='x2' weights w1,w2w1 = Value -3.0, label='w1' w2 = Value 1.0, label='w2' bias of the neuronb = Value 6.8813735870195432, label='b' x1 w1 + x2 w2 + bx1w1 = x1 w1; x1w1.label = 'x1 w1'x2w2 = x2 w2; x2w2.label = 'x2 w2'x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1 w1 + x2 w2' n = x1w1x2w2 + b; n.label = 'n'o = n.tanh ; o.label = 'o' o.backward Below graph is the output of this code block NOTES : at this point of time grad of all the nodes will be zero. Everytime we perform math operations like +, -, , /, , tanh , exp , this class builds a tracking network in our computer’s memory. Activation Function using tanh but reLu is used in industry As we discussed that activation function quashes the output into a specific range like 0 to 1, or -1 to 1 , it introduce non-linearity to the network. They break the chain of linear transformations, preventing the multi-layer martix structure from collapsing into a single linear equation, which enable the mode to approximate complex high- dimesional decision boundaries. Embedding Local Calculus Rules Calculating Gradient - Here you will understand why we need to store the nodes. So once your forward calculations work seamlessly and build this graph, you will see that with the inputs, weights, biases and the activation function our network has generated a prediction, But right now, that prediction is just a raw number floating in memory. The network has no idea if its guess is incredibly accurate or completely wrong. But in this example we don’t have the target output, just for understanding the core algorithm. Now every mathematical gate in our code needs to know how to calculate its own derivative when we decide to look backward.To http://backward.to/ do this, inside methods like add , mul , tanh, and pow , we nest a secret internal function called backward . • The Local Rule: Each operation holds its own specific calculus rule like the addition rule or the product rule . • Passing the Blame: When triggered, this function takes the incoming error from the front out.grad , scales it using its local rule, and passes it backward to its parent nodes. • The Accumulator += : We use += instead of a simple = when updating gradients. This ensures that if a single variable is used in multiple different math problems at the same time like b = a + a , its error contributions from both paths add up correctly instead of wiping each other out. Finding the Path: Topological Sort build topo - Now, our graph is fully build, and every node is packed with its local calculus rules. We are ready to find our mistake gradient . But we run into a major roadblock: what order do we calculate things in? In a massive mathematical equation, you cannot just calculate derivatives at random. A parent node cannot safely calculate its gradient until its child node has completely finished resolving its upstream calculations. If we do this out of order, the chain rule collapses. We need a perfect, chronological checklist. Finding the Path: Topological Sort build topo To solve our ordering problem, we write a nested function called build topo. This is an algorithm that scans our expression tree to create a sequential timeline of dependencies. • How it explores: It starts at the final output and uses a programming technique called recursion to crawl backward through our self. prev memory map. • Building the Checklist: It strictly ensures that a node is only appended to our topo list after all of its parents have been completely visited and cleared. • The Result: This generates a perfectly ordered list from the absolute base inputs up to the final output node. Executing the Chain Rule The Grand Finale Now we have our checklist, and everything leads up to this final moment. The backward method is triggered, and the engine goes into reverse gear. • Setting the Anchor: We start by setting the final output node's gradient to 1.0. Mathematically, this just means a variable's rate of change with respect to itself is always 1 \ \frac{\partial L}{\partial L} = 1\ . • Flipping the List: We run a loop using reversed topo . This flips our timeline upside down, forcing the execution to flow flawlessly backward from the final answer down to the raw starting inputs. • Unleashing the Gradients: As the loop steps through each node in reverse order, it fires off that node's custom node. backward function that we tucked away earlier. One by one, like falling dominoes, every single gradient in our network gets populated automatically. Now that our atomic Value engine can track data, perform math operations, and calculate gradients automatically, we have all the raw materials we need. It is time to scale up. We will use our custom objects to build the structural layers of a Multi-Layer Perceptron MLP —the actual neural network architecture behind modern AI. The Single Neuron Class class Neuron The fundamental building block of our network is a single neuron. python classNeuron:def init self,nin : self.w = Value random.uniform -1,1 for in range nin self.b = Value random.uniform -1,1 Stacking Neurons: The Layer Class class Layer A single neuron can only do so much. To learn complex mathematical boundaries, we stack multiple neurons side-by-side to create a network layer. python classLayer:def init self,nin,nout : self.neurons = Neuron nin for in range nout The Complete Blueprint: The Multi-Layer Perceptron class MLP Finally, we chain multiple layers sequentially on top of each other to form a complete deep neural network. python classMLP:def init self,nin,nouts :sz = nin + nouts self.layers = Layer sz i , sz i+1 foriin range len nouts To see our newly engineered architecture actually learn, we feed it a classic multi-input toy dataset. python A dataset of 4 separate samples, each containing 3 input featuresxs = 2.0,3.0, -1.0 , 3.0, -1.0,0.5 , 0.5,1.0,1.0 , 1.0,1.0, -1.0 , The actual ground-truth target answers we want the AI to predictys = 1.0, -1.0, -1.0,1.0 n = MLP 3, 4,4,1 3 inputs, two hidden layers of 4 neurons, 1 final output neuron Now, we write an optimization loop to run our cyclical pipeline over and over again for 20 epochs iterations . This is where the magic happens: python forkin range 20 : Step A: The Forward Pass python ypred = n x for xin xs loss = sum yout - ygt 2for ygt, youtin zip ys, ypred Step B: The Backward Pass Backpropagation python for pin n.parameters : p.grad =0.0 loss.backward Step C: The Optimization Tweak Gradient Descent python for pin n.parameters : p.data += -0.1 p.grad The Result python print k, loss.data And that is the magic of autograd. We started with the basic math of limits and derivatives, built a simple tracking container from scratch, and scaled it up into a network of neurons that can actually think and adapt.Every time you prompt ChatGPT, Claude, or Gemini, this exact cyclical pipeline is running at a massive scale behind the scenes: a forward pass to predict, a loss function to grade, a backward pass to find mistakes, and a slight tweak to fix them. I use to find deep learning very intriguing I still do, but not in the same way because before learning this I used to thing that there is some magic by this I means a very very complex thing that I wont be able to comprehend going underneath, but underneath all the hype, it is just code, calculus, and a system built to learn from its own errors. Foundation of the of Deep Learning https://pub.towardsai.net/foundation-of-the-of-deep-learning-471f5b1e1698 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.