How Neural Networks Actually Work — A Thread for Curious Minds

A developer explained that neural networks are built from stacked linear equations with non-linear activation functions, using gradient descent and backpropagation to minimize loss. The post breaks down the mathematical foundations, from basic line equations to the Universal Approximation Theorem, showing how weighted connections and activation functions enable networks to model complex patterns.

Everything starts from something you already know: y = mx + c That's just a line. But stack enough of them, connect them, and add non-linearity? You have a neural network. Here's the full breakdown ━━━━━━━━━━━━━━━ 📌 TRAINING — How the Model Learns We don't know the best values of m and c at first. So we: L = y − ŷ ² This is Mean Squared Error MSE . Our goal? Minimize this loss. ━━━━━━━━━━━━━━━ 📐 Gradients — The Learning Step We use differentiation to see how changing m or c affects the loss. These are called gradients. Then we use gradient descent: m new = m old − η · ∂L/∂m c new = c old − η · ∂L/∂c Where η = learning rate how fast the model updates . ━━━━━━━━━━━━━━━ 🔗 From Line to Neural Network Now imagine multiple inputs — x₁, x₂, x₃... y = w₁x₁ + w₂x₂ + w₃x₃ + ... + b → wᵢ = weight for each input how important that input is → b = bias like c, helps shift the curve Each xᵢ, wᵢ pair = one "connection strength." This is one neuron. ━━━━━━━━━━━━━━━ 🏗️ The Network Structure → Input Layer: where data enters x1, x2, x3... → Hidden Layers: learn complex features → Output Layer: gives the final prediction Each neuron connects to neurons in the next layer. Every connection has its own weight. Output of each neuron = f W · X + b ━━━━━━━━━━━━━━━ ⚡ Activation Functions — Adding Non-Linearity If we combine weighted inputs linearly, the model can only learn straight lines. Real-world data is non-linear — so we add activation functions: • Sigmoid → probabilities 0 to 1 • ReLU → max 0, x — adds non-linearity, efficient • Tanh → centered around 0 • Softmax → multi-class classification These allow the network to model complex, curved decision boundaries. ━━━━━━━━━━━━━━━ 🌐 Universal Approximation Theorem This is the heart of deep learning. "A neural network with enough neurons and layers can approximate any function in the world — no matter how complex — as long as you have enough data and training." Translation: They can model any pattern, from stock prices to language semantics. ━━━━━━━━━━━━━━━ 🔢 Why Matrices? Instead of computing one weight at a time, we represent inputs, weights, and biases as matrices: Y = f WX + b This allows vectorized computation — very fast on GPUs. ━━━━━━━━━━━━━━━ 🔁 Backpropagation — Learning in Multi-Layer Networks When you have many layers: That's backpropagation — the backbone of neural network training. ━━━━━━━━━━━━━━━ 🧾 Key Concepts Summary • Weights W → strength of connection between neurons • Bias b → shifts decision boundary • Activation Function → adds non-linearity • Loss Function → measures error • Gradient Descent → minimizes loss by adjusting weights • Backpropagation → passes errors backward ━━━━━━━━━━━━━━━ 🔄 The Visual Flow: Input Layer → Hidden Layer s → Output Layer → Weighted Sum → Activation → Loss Computation → Backpropagation → Update Weights Repeat until the network learns patterns perfectly. ━━━━━━━━━━━━━━━ 🌍 Real-World Analogy Think of it like how humans learn: • Inputs = sensory data • Weights = attention/importance we give each input • Bias = our default tendency • Activation = whether our brain reacts or not • Loss = how wrong we were • Gradients = how we adjust next time ━━━━━━━━━━━━━━━ 💡 In Short: Neural networks = layers of weighted connections that transform input → output, learning to minimize loss through gradient-based optimization and non-linear activation.