I was always curious about algorithms running behind models like GPT, Claude, and Gemini, etc.
So I was researching what is the actual algorithm/engine or foundation that makes these models train and predict things.
As I was searching, I stumbled upon Andrej Karpathy’s “Autograd” and what
Foundation of Deep Learning -
I was always curious about the algorithms behind models like GPT, Claude, and Gemini, etc.So I was researching what actual algorithm or foundation is used that makes these models train themselves and predict things.As I was searching, I stumbled upon Andrej Karpathy’s “Autograd”. In this blog, I will try to cover about each and every thing I possibly can, so if you don’t know much about coding, you would also be able to understand if you are just curious enough.
The most basic working of Autograd is -
It is an automatic differentiation engine that actually calculates the derivative or slope of the mathematical operation in your code.
In non mathmatical term — it is an engine that calculates the exact adjustments needed to help the network learn and make predictions
So, what a neural network does is: it takes input (data), parameters(weights and biases), and it performs some mathematical operations (like multiplications and additions), stores all the calculations inside nodes, and makes a prediction; that is what we call a forward pass.
After it makes its first prediction, it calculates the Loss(how far the prediction is from the actual answer), and we call this a loss function.
Now, to make the model predict the correct one by minimizing the gap between the prediction and the actual answer i.e loss, it triggers backpropagation, reading its hidden history map (nodes) in reverse and finding gradients, which means showing how much each input weight contributed to the total error.
Prerequisite Information -
What are weights?Weight controls the strength of the connection between two neurons. It tell/decides how much influence does the input have on the final output.
Working -
Every piece of the incoming data is multiplied by its corresponding weights.
The math = (Output = input * weight)
Example — House Price prediction, the number of bedrooms and the color of the front door are the input. The network will assign a heavy positive weight to bedroom because they matter a lot, and a neat-zero weight to the door color as it is irrelevant
What are biases?Bias is an extra number that added to the sum of your weighted inputs. It allows you to shift your entire calculation up or down, completely independent of the inputs.
Working -
It act like a intercept © in the classic line equation y = mx + c.
Even if all the inputs are zero, the biases ensures the neuron can still output a value.
What is an activation function?Activation Function decided whether a neuron should “fire”(pass information forward) or not.
Working -
It takes the combined math of your inputs, weights, and biases (input * weight + bias) and squashes it into a specific range (like 0 to 1, or -1 to 1)
without this, ai is just a giant calculator adding and multiplying numbers. It can only draw straight lines. It could never understand a curved pattern.
Example -
In house price prediction, a house cannot have a negative price. An activation function (like ReLU) will automatically turn any negative calculation into a clean zero, keeping the data realistic.
**What is Loss function?**Loss Function is the network’s report card. It measures the error by calculating exaclty how far off the network’s prediction was from the actual true answer.
Working -
It takes the final output of your network that it predicted and compares it to the actual target /answer.
Example -
If the network predicts a house costs $300k, but it actually costs $400k, the loss function calculates that $100k gap and converts it into a single “error score” for Autograd to look at.
What is gradient or derivative?Gradient in simple terms means rate of change. it measures the change happened at every node of the mathematical operation, denoting exactly how much that specific node is responsible for the network’s final error
Working-
It will twl you the slope of the loss function. If the gradient is +ve increasing the weight will increase the error. If it is -ve, increasing the weight will decrease the error.
Example-
Think of being blindfolded on a mountain (the mountain is the error). The gradient tells you which direction is “downhill” so you can safely walk toward the bottom (zero error).
Actual flow -
input → input integrating with paramters(weights and biases) going inside a neural network → output the prediction → find the loss → twike the numbers to minimize the error → repeat till prediction == target
Understanding the code -
class Value:
python
def __init__(self, data, children=(), _op='', label = ''):
self.data = data
self._prev = set(children) self._op = _op self.grad = 0.0 self._backward = lambda: None self.label = label
python
def __repr__(self): return f"Value(data = {self.data})"
python
def __neg__(self): return self * -1
python
def __sub__(self, other): return self + (-other)
python
def __add__(self, other): other = other if isinstance(other, Value) else Value(other) out = Value(self.data + other.data, children=(self, other), _op='+')
python
def _backward(): self.grad += 1.0 * out.grad other.grad += 1.0 * out.grad
out._backward = _backward return out
python
def __mul__(self, other): other = other if isinstance(other, Value) else Value(other) out = Value(self.data * other.data, children = (self, other), _op='*')
python
def _backward(): self.grad += other.data * out.grad other.grad += self.data * out.grad
out._backward = _backward return out
python
def __rmul__(self, other): return self * other
python
def __truediv__(self, other): return self * (other ** -1)
python
def __radd__(self, other): # other + self return self + other
python
def tanh(self): x = self.data t = (math.exp(2*x) - 1) / (math.exp(2*x) + 1) out = Value(t, children = (self, ), _op='tanh')
python
def _backward(): self.grad += (1 - t**2) * out.grad
out._backward = _backward return out
python
def exp(self): x = self.data out = Value(math.exp(x),(self, ), 'exp')
python
def _backward(): self.grad += out.data * out.grad
out._backward = _backward return out
python
def __pow__(self, other): assert isinstance(other, (int, float)) out = Value(self.data ** other, (self, ), f"**{other}")
python
def _backward(): self.grad += other * (self.data ** (other -1)) * out.grad
out._backward = _backward return out
python
def backward(self): topo = [] visited = set()
python
def build_topo(v): if v not in visited: visited.add(v) for child in v._prev: build_topo(child) topo.append(v)
build_topo(self)
self.grad = 1 for node in reversed(topo): node._backward()
So this is our Value object, the automatic differentiation engine.
It is the core mathematical Backbone needed to train a Neural Network.
Starting with -
Methods / Functions for Mathematical operations — creating methods
We write methods like add and mul to trick Python into using math symbols on custom objects. It includes type checking (isinstance) so mixing regular numbers (like a + 2) doesn't cause a crash. It also recycles existing code: subtraction (sub) is re-engineered as adding a negative number, and division (truediv) is re-engineered as multiplying by a negative power.
Building a Computational Graph (Forward Pass)
n = x1w1x2w2 + b; n.label = 'n'o = n.tanh(); o.label = 'o'
o.backward()
Below graph is the output of this code block
NOTES : at this point of time grad of all the nodes will be zero.
Everytime we perform math operations (like +, -, *, /, **, tanh(), exp()), this class builds a tracking network in our computer’s memory.
Activation Function ( using tanh() but reLu is used in industry)
As we discussed that activation function quashes the output into a specific range (like 0 to 1, or -1 to 1), it introduce non-linearity to the network. They break the chain of linear transformations, preventing the multi-layer martix structure from collapsing into a single linear equation, which enable the mode to approximate complex high- dimesional decision boundaries.
Embedding Local Calculus Rules ( Calculating Gradient ) -
Here you will understand why we need to store the nodes.
So once your forward calculations work seamlessly and build this graph, you will see that with the inputs, weights, biases and the activation function our network has generated a prediction, But right now, that prediction is just a raw number floating in memory. The network has no idea if its guess is incredibly accurate or completely wrong. But in this example we don’t have the target output, just for understanding the core algorithm.
Now every mathematical gate in our code needs to know how to calculate its own derivative when we decide to look backward.To do this, inside methods like add, mul, tanh, and pow, we nest a secret internal function called _backward().
• The Local Rule: Each operation holds its own specific calculus rule (like the addition rule or the product rule).
• Passing the Blame: When triggered, this function takes the incoming error from the front (out.grad), scales it using its local rule, and passes it backward to its parent nodes.
• The Accumulator (+=): We use += instead of a simple = when updating gradients. This ensures that if a single variable is used in multiple different math problems at the same time (like b = a + a), its error contributions from both paths add up correctly instead of wiping each other out.
Finding the Path: Topological Sort (build_topo) -
Now, our graph is fully build, and every node is packed with its local calculus rules. We are ready to find our mistake(gradient). But we run into a major roadblock: what order do we calculate things in?
In a massive mathematical equation, you cannot just calculate derivatives at random. A parent node cannot safely calculate its gradient until its child node has completely finished resolving its upstream calculations. If we do this out of order, the chain rule collapses. We need a perfect, chronological checklist.
**Finding the Path: Topological Sort (**build_topo)
To solve our ordering problem, we write a nested function called build_topo. This is an algorithm that scans our expression tree to create a sequential timeline of dependencies.
• How it explores: It starts at the final output and uses a programming technique called recursion to crawl backward through our self._prev memory map.
• Building the Checklist: It strictly ensures that a node is only appended to our topo list after all of its parents have been completely visited and cleared.
• The Result: This generates a perfectly ordered list from the absolute base inputs up to the final output node.
Executing the Chain Rule (The Grand Finale)
Now we have our checklist, and everything leads up to this final moment. The backward() method is triggered, and the engine goes into reverse gear.
• Setting the Anchor: We start by setting the final output node's gradient to 1.0. Mathematically, this just means a variable's rate of change with respect to itself is always 1 ((\frac{\partial L}{\partial L} = 1)).
• Flipping the List: We run a loop using reversed(topo). This flips our timeline upside down, forcing the execution to flow flawlessly backward from the final answer down to the raw starting inputs.
• Unleashing the Gradients: As the loop steps through each node in reverse order, it fires off that node's custom node._backward() function that we tucked away earlier. One by one, like falling dominoes, every single gradient in our network gets populated automatically.
Now that our atomic Value engine can track data, perform math operations, and calculate gradients automatically, we have all the raw materials we need. It is time to scale up. We will use our custom objects to build the structural layers of a Multi-Layer Perceptron (MLP)—the actual neural network architecture behind modern AI.
**The Single Neuron Class (**class Neuron)
The fundamental building block of our network is a single neuron.
python
classNeuron:def__init__(self,nin): self.w = [Value(random.uniform(-1,1))for_in range(nin)] self.b = Value(random.uniform(-1,1))
**Stacking Neurons: The Layer Class (**class Layer)
A single neuron can only do so much. To learn complex mathematical boundaries, we stack multiple neurons side-by-side to create a network layer.
python
classLayer:def__init__(self,nin,nout): self.neurons = [Neuron(nin)for_in range(nout)]
**The Complete Blueprint: The Multi-Layer Perceptron (**class MLP)
Finally, we chain multiple layers sequentially on top of each other to form a complete deep neural network.
python
classMLP:def__init__(self,nin,nouts):sz = [nin] + nouts self.layers = [Layer(sz[i], sz[i+1])foriin range(len(nouts))]
To see our newly engineered architecture actually learn, we feed it a classic multi-input toy dataset.
python
Now, we write an optimization loop to run our cyclical pipeline over and over again for 20 epochs (iterations). This is where the magic happens:
python
forkin range(20):
Step A: The Forward Pass
python
ypred = [n(x)for xin xs] loss = sum((yout - ygt)**2for ygt, youtin zip(ys, ypred))
Step B: The Backward Pass (Backpropagation)
python
for pin n.parameters(): p.grad =0.0 loss.backward()
Step C: The Optimization Tweak (Gradient Descent)
python
for pin n.parameters(): p.data += -0.1 * p.grad
The Result
python
print(k, loss.data)
And that is the magic of autograd. We started with the basic math of limits and derivatives, built a simple tracking container from scratch, and scaled it up into a network of neurons that can actually think and adapt.Every time you prompt ChatGPT, Claude, or Gemini, this exact cyclical pipeline is running at a massive scale behind the scenes: a forward pass to predict, a loss function to grade, a backward pass to find mistakes, and a slight tweak to fix them.
I use to find deep learning very intriguing I still do, but not in the same way because before learning this I used to thing that there is some magic (by this I means a very very complex thing that I wont be able to comprehend) going underneath, but underneath all the hype, it is just code, calculus, and a system built to learn from its own errors.
Foundation of the of Deep Learning was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.