{"slug": "before-i-could-build-gpt-i-had-to-teach-a-machine-to-learn", "title": "Before I Could Build GPT, I Had to Teach a Machine to Learn", "summary": "A JavaScript developer documented building a neural network from scratch in plain JavaScript—no TensorFlow or PyTorch—to understand how language models learn, starting with a supervised-learning example that predicts the sum of two numbers. The engineer implemented a single neuron as a weighted sum plus bias, then a squared-error loss function to measure prediction error, establishing the foundation for gradient-based training before approaching GPT-scale models.", "body_md": "30 July 2026.\n\nI didn't start this journey because I wanted to build another AI chatbot.\n\nI started because I was using AI tools regularly, and I realised something slightly uncomfortable: I knew how to use them, but I couldn't properly explain what was happening underneath.\n\nI could ask ChatGPT to write code, explain a concept, or help debug something. But if someone asked me, \"How does a language model actually learn?\", I would have struggled to give an answer beyond a few familiar terms.\n\nNeural networks. Training data. Transformers. Attention. Weights.\n\nI knew the vocabulary. I wanted to understand the machinery.\n\nAs a JavaScript developer, I also had a preference: I wanted to build things myself. I didn't want to import a library, call a `train()` function and pretend I understood what happened.\n\nSo I set myself a challenge:\n\nCould I understand the fundamentals of AI by implementing them in JavaScript, from scratch?\n\nNo TensorFlow. No PyTorch. No machine-learning framework hiding the mathematics. Just JavaScript, arrays, loops, and a lot of patience.\n\nI had no idea how many times I would end up staring at a number that had suddenly become `NaN`.\n\nBut that was part of the journey.\n\nBefore getting anywhere near GPT, I needed to understand a much smaller problem.\n\nImagine I give a program two numbers:\n\n``` js\nconst x1 = 2;\nconst x2 = 3;\n```\n\nAnd I want it to predict their sum.\n\nThe answer is obviously 5. I can write:\n\n``` js\nconst result = x1 + x2;\n```\n\nBut that isn't machine learning. I've already told the program exactly what to do.\n\nWhat if I don't tell it the relationship? What if I only give it examples?\n\n| Input 1 | Input 2 | Expected output | \n|---|---|---|\n| 2 | 3 | 5 | \n| 4 | 1 | 5 | \n| 5 | 5 | 10 | \n| 7 | 2 | 9 | \n\nNow the program needs to discover the relationship from the examples.\n\nThis is the basic idea behind supervised learning: provide examples with known answers, make predictions, measure the errors, and adjust the model so that future predictions improve.\n\n*Supervised learning is a loop: show the model examples with known answers, make a prediction, compare it with the expected output, and adjust the weights.*\n\nThe important change in thinking was this:\n\nInstead of programming the answer, I was programming a system that could learn how to produce the answer.\n\nThat distinction became the foundation of everything that followed.\n\nA neuron is a small mathematical operation.\n\nIt receives inputs, multiplies them by weights, adds a bias, and produces an output.\n\nFor two inputs:\n\nHere:\n\n*A neuron scales its inputs by weights, adds a bias to get the weighted sum z, then applies an activation f to produce the output ŷ.*\n\nLet's implement it.\n\n```\nfunction neuron(x1, x2, w1, w2, bias) {\n  return x1 * w1 + x2 * w2 + bias;\n}\n\nconst prediction = neuron(\n  2,\n  3,\n  0.5,\n  0.8,\n  0.1\n);\n\nconsole.log(prediction);\n```\n\nThe weights determine how strongly each input contributes to the result. The bias lets the model shift the output.\n\nAt first, these values can be random. The model doesn't know the correct relationship yet.\n\nAnd that leads to the next question:\n\nHow does it know whether its prediction is good or bad?\n\nSuppose the expected answer is 5, but our neuron predicts 4.\n\nWe need a way to measure that difference.\n\nOne simple option is squared error:\n\nWhere:\n\nIn JavaScript:\n\n``` js\nfunction loss(prediction, expected) {\n  const difference = prediction - expected;\n  return difference * difference;\n}\n\nconsole.log(loss(4, 5)); // 1\nconsole.log(loss(2, 5)); // 9\n```\n\nA prediction of 2 produces a larger loss than a prediction of 4.\n\n*Squared error L = (ŷ − y)² is a parabola centred on the expected answer. ŷ = 4 gives L = 1, ŷ = 2 gives L = 9, and the loss is zero only when ŷ = y = 5.*\n\nThis gives us a measurable objective: reduce the loss.\n\nI found this idea surprisingly important. A model doesn't simply \"know\" that it made a mistake. We define a mathematical function that measures how wrong its prediction was.\n\nAnd then we need to use that measurement to improve the model.\n\nImagine the loss function as a landscape.\n\nOur weights determine where we stand on that landscape. Some combinations produce a high loss; others produce a lower loss.\n\nGradient descent tells us which direction to move the weights to reduce the loss.\n\nThe basic update rule is:\n\nHere, η is the learning rate.\n\nThe derivative tells us how sensitive the loss is to a particular weight. The learning rate controls how large an update we make.\n\n*Gradient descent steps downhill on the loss. A well-chosen learning rate converges toward the minimum; one that is too large overshoots and the steps grow.*\n\nIn code, the update itself is straightforward:\n\n```\nfunction updateWeight(weight, gradient, learningRate) {\n  return weight - learningRate * gradient;\n}\n```\n\nThe difficult part is calculating the gradient correctly.\n\nThat led me to one of the most important concepts in the entire journey: backpropagation.\n\nLet's return to our neuron:\n\nAnd use squared error:\n\nWe want to know how changing w1 , w2 , or b affects the loss.\n\nUsing the chain rule:\n\nSimilarly:\n\nAnd:\n\nThese gradients tell us how each parameter contributed to the error.\n\n*The forward pass computes ŷ and the loss. The backward pass applies the chain rule so that ∂L/∂w₁ = 2(ŷ − y)x₁, and likewise for w₂ and b.*\n\nThis is the intuition behind backpropagation: calculate how the error at the output depends on the intermediate calculations and parameters, then pass that information backwards through the network.\n\nIt isn't magic. It's calculus applied systematically.\n\nOnce I understood that, neural networks became much less mysterious.\n\nA single neuron is useful for understanding the basic mathematics, but real models need many neurons connected across layers.\n\nI moved from simple linear models to a trainable multilayer perceptron.\n\nOne of the architectures I worked with was:\n\n```\n2 inputs\n   |\n   v\n4 neurons\n   |\n   v\nActivation function\n   |\n   v\n1 output\n```\n\nThe network could learn a simple relationship such as:\n\n*A 2–4–1 multilayer perceptron: two inputs, four hidden neurons with an activation function, and one output, fully connected layer to layer.*\n\nThe architecture introduced another important idea: activation functions.\n\nWithout suitable nonlinear activations, stacking linear transformations still produces an overall linear transformation. That limits what the network can learn.\n\nSo I explored several activation functions.\n\nSimple, fast, and commonly used.\n\nProduces values between -1 and 1.\n\nUnlike ordinary ReLU, it allows a small negative slope for negative inputs.\n\nThis helps avoid neurons that stop producing useful gradients for negative inputs.\n\nGELU is smoother than ReLU and is commonly used in Transformer architectures.\n\n*The four activation functions I explored. Without a nonlinearity, stacking layers would still be a single linear transformation.*\n\nI implemented GELU myself and checked its derivative numerically.\n\nThat brought me to another lesson.\n\nWhen implementing backpropagation manually, a small mathematical mistake can quietly ruin training.\n\nA gradient might be incorrect. The model might still run. The loss might even change. But the learning behaviour could be wrong.\n\nSo I implemented numerical gradient checking.\n\nThe idea is simple.\n\nIf we slightly increase a parameter, calculate the loss, then slightly decrease it and calculate the loss again, we can estimate its derivative:\n\n*Numerical gradient checking compares a secant through L(w − ε) and L(w + ε) with the analytic tangent from backpropagation. If they agree, the derivative is likely correct.*\n\nThen we compare that numerical estimate with the gradient produced by our backpropagation implementation.\n\nIf they are sufficiently close, that's evidence that our derivative implementation is correct.\n\nThis became one of my favourite debugging techniques because it gave me a way to test the mathematics rather than just hoping the code was right.\n\nNot everything worked smoothly.\n\nThere were exploding gradients. There were `NaN` values. There were dead neurons. There were situations where a network would train for a little while and then produce completely unusable numbers.\n\n*Loss over epochs for three runs: healthy training, exploding gradients, and dead neurons that flatten out and then become NaN.*\n\nOne of the lessons was that a correct-looking architecture doesn't guarantee stable training.\n\nThe learning rate matters. Weight initialization matters. Activation functions matter. Gradient calculations matter.\n\nA model can fail for reasons that aren't obvious from looking at the final output.\n\nAnd that's why implementing these pieces myself was so useful. I wasn't just seeing that training failed. I could inspect the values and understand where the failure might be happening.\n\nBy this point, I had a much clearer picture of how a basic neural network learns.\n\nThe process looks something like this:\n\nThat simple loop is an important foundation for understanding much larger models.\n\nBut there was still a huge gap between a small neural network that learns to add two numbers and a language model that can generate paragraphs.\n\nA language model has to process sequences. It needs to represent text, understand relationships between tokens, and predict what comes next.\n\nThat meant I was ready to move beyond ordinary neural networks.\n\nI had learned how to teach a machine to learn. Now I wanted to understand how a machine learns language.\n\n**Coming next: Part 2 — The Transformer: How GPT Learns to Pay Attention.**\n\nIn the next article, I'll explore tokenization, embeddings, positional information, Query-Key-Value projections, self-attention, multi-head attention, residual connections and LayerNorm — the building blocks that eventually led to our own GPT implementation.\n\nIf you'd like to follow along, subscribe to the series so you don't miss Part 2.\n\n*Project journey started: 30 July 2026.*", "url": "https://wpnews.pro/news/before-i-could-build-gpt-i-had-to-teach-a-machine-to-learn", "canonical_source": "https://dev.to/roshanshikhare/before-i-could-build-gpt-i-had-to-teach-a-machine-to-learn-4mac", "published_at": "2026-11-11 05:41:21+00:00", "updated_at": "2026-11-11 05:51:05.828178+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "artificial-intelligence", "large-language-models"], "entities": ["JavaScript", "TensorFlow", "PyTorch", "GPT", "ChatGPT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/before-i-could-build-gpt-i-had-to-teach-a-machine-to-learn", "markdown": "https://wpnews.pro/news/before-i-could-build-gpt-i-had-to-teach-a-machine-to-learn.md", "text": "https://wpnews.pro/news/before-i-could-build-gpt-i-had-to-teach-a-machine-to-learn.txt", "jsonld": "https://wpnews.pro/news/before-i-could-build-gpt-i-had-to-teach-a-machine-to-learn.jsonld"}}