Neural networks work by building a system inspired by the human brain that can learn how to solve the problem by looking at examples.If you are new to machine learning, neural networks might seem like an impenetrable black box full of dense mathematics. However, the core idea behind them is surprisingly simple, intuitive, and beautiful.
In this guide, we will unpack neural networks from first principles, stripping away the hype to see how they truly work.
Mathematically, a neural network is just a giant, adjustable mathematical function.
A neural network is a tool for Function Approximation. It begins as a blank slate—a mathematical machine full of adjustable "knobs" and "dials." Initially, these dials are set randomly, meaning the network gives terrible guesses. But by feeding it thousands of examples (inputs paired with correct outputs), we gradually tweak those knobs until the network’s output matches the desired result.In essence: A neural network learns to approximate the invisible mathematical rule that connects inputs to outputs.
The name "Neural Network" comes from biological neuroscience. Our brains contain approximately 86 billion interconnected biological cells called neurons.Biological Neuron Flow:
[ Dendrites ] ---> ( Cell Body ) ---> [ Axon ] ---> Synapses (Processing) (Output) (Connections)
Dendrites receive electrical signals from other neurons.The Cell Body aggregates these incoming signals. If the combined electrical charge crosses a certain threshold, the neuron "fires". The Axon carries the fired signal down toward the end of the cell.Synapses transmit the signal across junctions to neighboring neurons.In artificial intelligence, we do not build actual biological structures. Instead, we create a mathematical simplified abstraction called an Artificial Neuron (or a Perceptron).
To understand how a complex deep network functions, we must first understand its smallest building block: a single artificial neuron.
An artificial neuron performs four fundamental operations:
Receives Inputs, Multiplies each input by a Weight, Adds a constant called Bias, Passes the result through an Activation Function, to produce the final Output.
Let’s break down each of these components in plain language:
A. Inputs ($x$) Inputs represent the features of the data you want to analyze.
B. Weights ($w$) Weights are the "knobs and dials" of the network. They determine how much influence a given input has on the neuron’s final decision: A large positive weight means the input strongly drives the decision upward. A weight close to zero means the input is mostly ignored. A negative weight means as the input increases, the prediction goes down.
Mathematically, we multiply each input by its weight
C. Bias ($b$) The Bias is an extra adjustable parameter added to the weighted sum. It acts as a base threshold, shifting the calculation up or down regardless of the inputs.
It gives the neuron the freedom to trigger even when inputs are low, or remain quiet even when inputs are high.Adding the bias gives us the linear equation:
$z = \left( \sum_{i=1}^{n} w_i x_i \right) + b$
D. Activation Function ($f(z)$)
If a neural network consisted only of weights and biases, it would just be doing linear algebra—drawing straight lines through data. But real-world data is non-linear, full of curves, twists, and subtle thresholds.
The Activation Function introduces non-linearity into the network. It decides whether the neuron should activate and by how much.
Here are three common activation functions:
1. The Step FunctionHistorically, early neurons used a binary switch: if $z > 0$, output $1$; otherwise output $0$. This mirrors a simple on/off switch, but it lacks nuance because small changes in inputs can cause sudden, wild jumps in output.
2. The Sigmoid Function The Sigmoid function squashes any real number input into a smooth range between 0 and 1:
$\sigma(z) = \frac{1}{1 + e^{-z}}$
. It is often used when predicting probabilities
3. ReLU (Rectified Linear Unit) Despite its fancy name, ReLU is remarkably simple:
$f(z) = \max(0, z)$
If $z$ is negative, output 0. If $z$ is positive, return $z$ as it is.
Because it is computationally efficient and works incredibly well, ReLU is the most popular activation function used inside modern neural networks today.
4. Tanh(Hyperbolic Tangent)
$\tanh(x) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$
5. Leaky ReLU
$f(x) = \max(\alpha x, x)$
6. SoftMax
$\sigma (\mathbf{z})_{i}=\frac{e^{z_{i}}}{\sum _{j=1}^{K}e^{z_{j}}}$
Output range is between 0 and 1 and they sum up to 1.
A single neuron can make basic linear decisions (like drawing a straight line to divide two groups of points). However, real intelligence requires combining hundreds, thousands, or millions of these neurons into a network.A typical neural network is structured in vertical slices called Layers:
1. The Input Layer: This is where raw data enters the network. It doesn't perform calculations; it simply passes feature values forward.
2. The Hidden Layer: Located between input and output, these layers are called "hidden" simply because their intermediate processing values are not directly observed in the dataset.
Early hidden layers learn simple, fundamental patterns (e.g., detecting horizontal, vertical, or diagonal edges in an image).
Middle hidden layers combine those basic edges to detect shapes (e.g., circles, corners, texture patterns).
Deeper hidden layers combine shapes into high-level concepts (e.g., eyes, noses, wheels, or ears).
When a network has multiple hidden layers stacked on top of each other, we call it a Deep Neural Network — hence the term Deep Learning.
3. The Output Layer: The final layer produces the network's answer.For a regression task (predicting a house price), it might contain 1 single neuron outputting a raw number like $350,000.
For a classification task (identifying an animal as a Cat, Dog, or Bird), it might contain 3 neurons, each outputting a probability score for its category.
When you initialize a neural network, its weights ($w$) and biases ($b$) are filled with random numbers. Learning happens through a continuous loop consisting of four major steps:
Step 1: Forward Propagation
During forward propagation, data flows in one direction—from input to output: Inputs are fed into the input layer. Neurons calculate weighted sums, add biases, apply activation functions, and pass their results forward to the next layer.
This process continues layer by layer until the final output layer generates a prediction.
Step 2: Measuring Error (The Loss Function)
Once the network makes a prediction, we compare it against the real target answer ($y$) using a Loss Function (also called a Cost Function).
The Loss Function measures how wrong the network is.
A simple example is Mean Squared Error (MSE), commonly used for predicting numerical values:
$\text{Loss} = \frac{1}{2} (\hat{y} - y)^2$
The goal of training is simple: Adjust the network's weights and biases to make the Loss as close to zero as possible.
Step 3: Gradient Descent (Finding the Way Down)
Mathematically, the gradient is calculated using calculus (derivatives). It tells us two crucial pieces of information: Direction: Which way does the error increase or decrease?
Steepness: How fast is the error changing relative to changes in weight?
If we move our parameters in the opposite direction of the gradient, we walk downhill toward lower loss. The Learning Rate ($\eta$) The size of the step we take in each iteration is controlled by a hyperparameter called the Learning Rate ($\eta$): If the learning rate is too small: The network takes tiny steps and takes days or weeks to train.If the learning rate is too large: The network takes giant leaps and might overshoot the valley completely, bouncing wildly without ever learning.
Step 4: Backpropagation (Assigning Blame)
Backpropagation uses the mathematical Chain Rule from calculus to work backward through the network: Calculate the final error at the output layer, Determine how much the output layer weights contributed to that error, Pass the error metric back to the previous hidden layer, Determine how much that hidden layer's weights contributed and Repeat all the way back to the input layer.
By working backward, every single weight in the network receives a precise update instruction based on how much it contributed to the overall error.
While neural networks are extremely powerful, they are not without flaws. Working with them involves handling several distinct challenges:
1. Overfitting:
Overfitting happens when a network learns its training data too well. Instead of discovering general rules, it essentially memorizes the training examples, including their noise and random quirks.
2. Analogy:
A student who memorizes every practice question and answer key word-for-word, but fails completely on the actual test when questions are rephrased.
3. Solution:
Techniques like Regularization, Dropout (randomly disabling neurons during training), and collecting more diverse training data help prevent overfitting.
The Black Box Problem
When a traditional software system makes a decision, a developer can read the lines of code to understand why. But when a deep neural network with 175 billion weights makes a decision, it is nearly impossible for a human to look at those floating-point numbers and understand its exact reasoning. This lack of interpretability is a challenge in fields like healthcare, finance, and legal decision-making.
Data and Compute Requirements
Unlike simple statistical models, neural networks thrive on massive amounts of data. They also require significant computational power, often requiring specialized hardware like GPUs (Graphics Processing Units) or TPUs (Tensor Processing Units) to perform billions of matrix multiplications efficiently.
To consolidate what we've learned, here is a quick summary of the core pipeline:
| Concept | What It Is | Role in the Network |
| Neuron | Basic computational unit | Multiplies inputs by weights, adds bias, applies activation |
| Weight ($w$) | Adjustable strength factor | Controls how much importance an input has |
| Bias ($b$) | Base offset term | Allows the neuron to shift outputs up or down |
| Activation Function | Non-linear transform ($f$) | Enables network to learn complex, non-linear relationships |
| Forward Propagation | Data pass from input $\to$ output | Computes the network's current prediction |
| Loss Function | Error measurement | Quantifies how far off predictions are from truth |
| Backpropagation | Error distribution algorithm | Uses calculus to determine how to tweak each parameter |
| Gradient Descent | Optimization technique | Iteratively updates weights to reduce overall error |
A neural network is a system of connected mathematical units that learns patterns by adjusting numerical weights after making predictions and receiving feedback.
It receives information, processes that information through layers, makes a prediction, measures its error, and adjusts itself. Repeated enough times with good data, this process can produce systems capable of solving complex problems.
Neural networks are not copies of the human brain. They are tools built from mathematics, data, and computation. Their power comes from their ability to learn complicated patterns that would be difficult to describe using hand-written rules.