# Neural Networks: Weights, Activation, and Backpropagation

> Source: <https://dev.to/aahanchauhan/neural-networks-weights-activation-and-backpropagation-jd6>
> Published: 2026-09-06 22:51:56+00:00

If you've ever looked at a neural network diagram and every wondered what is actually happening in these networks, this post is just for you. I pulled this post together after researching the properties of neural networks, and writing out what is finally there to click. No analogies that lead to no explanations, just the math behind neural networks, one piece at a time.

Before neural networks got keep, they started shallow. A **perceptron** is a neural net with zero hidden layers, invented back in 1943. Perceptions work when data is *linearly separable*, meaning you could draw a line and cleanly split the classes on either side of it.

Let's take a simple example with an equation: D=y-2x-3. Everything where D>=0 is Class 1 and everything where D<0 is Class 2. This is basically how a model works: a single linear boundary that decides which side the example falls on.

However, real data is rarely this cooperative. The moment your classes tangle in a way no straight line can separate, you will need more than one layer. This is really where the deep learning begins.

Zoom into any single neuron in a network, and this is what entire computation occuring inside of it:

S = b∑ w_i * x_i

x_i are the inputs coming in, such as the pixel values and whatever your data is.

w_i are the weights (how much the network currently believes each input matters.

b is the bias that acts like a nudge that shifts the output independent of the inputs.

S is the logit, which is a unbounded value. The S value either shoots in the negative or positive direction indefinetely.

Stack raw logits like this across layers with nothing else added, and the whole network has a mathematical collapse into a linear function. Depth alone buys you nothing. You need a twist.

That very twist is the activiation function. This is applied to every logit before it moves onto the next layer. One of them is sigmoid:

ø(S) = 1 / (1+e)^-s

Just as the equation says, the ø(s) pushes the S value closest to 1 as the ø(s) value is positive. If ø(s) is negative, then the S value moves towards 0.

This nonlinearity is the backbone of how depth works. With nonlinearity, each added layer can fold the decision boundary into complex shapes, which is exactly what is required to separate tangled data.

A **layer** is just a bank of neurons, each computing its own weighted sum form the same inputs, and each with its own weights. The formula goes like this: 

z_i = ϕ (j∑ x_j * w_ij)

The middle layers are called the **hidden** layers. They are called hidden layers because the values are not directly obseved; only the input and the output (final value) are observed. The early layers tend to pick up the simple patterns while the deeper layers form abstract connection with the simple patterns made by the early layers.

The final output layer turns those hidden features into a prediction:

y_i = ϕ(j∑ z_j * w_ij)

For a K-way classification, the K output neurons are used. These neurons are used as one confidence score per class.

Say that the output layer displays three logits: (0.01, -3.8, 4.2). These numbers are all currently unbounded and do not hold any significant meaning at the moment. Softmax fixes this by taking any of those three logits and converting them into probabilities between 0 and 1 that sum up to exactly 1.0. The formula for softmax is:

p_j = (e^z_j) / (∑_k * e^zk)

In the example above, the 4.2 value would come out closer to a 90% or higher confidence score, while -3.8 will approach closer to 0, a lower confidence score.

One the network makes a prediction, there must be a way to measure how inaccurate it was. For the numerical predictions, it is normally considered the **Mean Squared Error**. For classification, it is almost always **Cross-Entropy**. This is how the formula goes:

CE = -n∑i=1 [y_i * log(p(yi)) + (1-yi) * log(1-p(yi))]

The intuition is quite simple: if the true label is 1 and the model confidently predicts closer to the O, the log term blows up and there are severe penalties. However, if the model was *correct*, the penalty shrinks towards zero. Cross-entropy rewards confident correctness and punishes the confident mistakes. This is exactly what you want out of a loss function.

Here's the loop that turns a "useless" network into a "useful" network:

That fifth step is called the **gradient descent**:

(w_ij)^new = w_ij - α * (∂E) / (∂w_ij)

α holds the **learning rate**.

Just picture the loss as a landscape. Training is descending that landscape, one gradient-calculated step at a time, trying to reach a low point. In a network with millions of billions of weights, that landscape has many dimensions with multiple valleys. This is part of why training is not guaranteed to find the *global* best solution, just a good one.

One pass through the *entire* training is called an **epoch**. Networks are trained across many epochs because each pass leaves weights in a sligtly different place. This is why seeing the same data again isn't wasted effort.

Due to restrictions, the code commensts will be in []. Here's a real and complete neural network trained on MNIST in PyTorch.

import torch

import torch.nn as nn

import torch.optim as optim

from torchvision import datasets, transforms

[Load Data]

convertToTensor = transforms.ToTensor()

train_data = datasets.MNIST(root='./data', train=True, download=True, transform=convertToTensor)

train_loader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)

[Define the network: 784 input pixels -> 128 hidden neurons -> 10 output classes]

model = nn.Sequential(

    nn.Flatten(),

    nn.Linear(28*28, 128),

    nn.ReLU(),

    nn.Linear(128, 10)

)

criterion = nn.CrossEntropyLoss()

optimizer = optim.SGD(model.parameters(), lr=0.01)

[Train]

for epoch in range(5):

    for images, labels in train_loader:

        optimizer.zero_grad()

        predictedLabels = model(images)

        loss = criterion(predictedLabels, labels)

        loss.backward()

        optimizer.step()

    print(f"Epoch {epoch}, Loss: {loss.item():.4f}")

Every concept from this post is sitting amongst these lines of code: nn.linear is the weighted sum, nn.reLU is the activation function, criterion computers cross-entropy loss, loss.backward() runs backpropagation, and optimizer.step() is the gradient descent update. This is basically just theory executed.

A neural network is a stack of weighted sums broken up by non-linear activations, trained by repeatedly measuring how wrong its predictions are and nudging every weight a tiny bit in the direction that makes it less wrong — and that exact loop, scaled up, is what's running behind every AI tool you've ever used.