Fintex on X: "The Math of Why Bigger AI Keeps Getting Smarter (Scaling Laws)

AI scaling laws, first published by OpenAI in January 2020, show that model performance improves predictably with scale, contradicting the narrative of unpredictable AI progress. The laws, which hold across seven orders of magnitude, reveal that each doubling of compute yields only a small improvement, driving the industry's race to add zeros. This predictability allows labs to forecast a model's capability before spending hundreds of millions of dollars, as demonstrated by the cost equation 6 × parameters × training tokens, which accurately predicted GPT-3's compute requirements.

I'm @ yusufknl - AI researcher by day, and I break down the math the hype skips. If this is your kind of thing, the follow is the move: I post this stuff weekly. DMs open for collabs and suggestions. Let's get straight to it. Everyone treats each new AI as a shock that fell out of the sky. GPT-3 stunned people in 2020, GPT-4 stunned them again in 2023, and every release since gets the same word attached: unpredictable. The story we're told is that these systems are so complex, so alien, that even their creators are surprised by what comes out. It makes for great headlines and better fundraising. Here's the secret the labs don't put on the launch livestream: it was all deeply predictable, and they had the graph years in advance. The single most important equation in modern AI isn't attention, and it isn't buried inside the model's weights. It's the one that tells you how good a model will be before you spend a hundred million dollars building it. It's called a scaling law, and it is, quite literally, a straight line on a graph. Once you see that line, the whole industry stops looking like a priesthood conjuring intelligence from silicon and starts looking like a room full of people reading a ruler. Let me show you the ruler. Checkpoint 01 The Straight Line Nobody Believes at First Start with the claim that sounds impossible: you can predict how smart an AI will be before you build it, the way an engineer predicts how much weight a bridge holds before pouring the concrete. In January 2020, a team at OpenAI led by Jared Kaplan and Sam McCandlish published Scaling Laws for Neural Language Models, and it turned that impossible claim into arithmetic They found that a model's error - its loss, the cross-entropy "surprise in bits" - falls as a smooth power law in three things: parameters, training tokens, and compute. Not one of them roughly. All three, cleanly. And it held across more than seven orders of magnitude. That's the gap between a firecracker and a nuclear bomb, in scale - and across all of it, the points fell on one straight line. In a field where almost nothing is clean, someone had found a law of physics. A power law looks like a curve on ordinary paper: But take the log of both sides and it becomes a straight line: So "the loss follows a scaling law" just means: put loss and compute on log axes, and the dots line up under a ruler. The slope is a single number - and you can measure it on tiny models, then extend the line to predict a model a thousand times bigger. Here it is in a dozen lines. Synthetic data, but exactly what the labs do with real training curves: Now the twist that turned this into a fifteen-year spending war. The exponents are tiny - around 0.05 to 0.10. A small exponent is cruel: each doubling of scale buys only a small, fixed sliver of improvement. The line is straight, but it's shallow. The only way to move a meaningful distance along a shallow line is to multiply your inputs by ten, then ten again. That single fact is why the whole industry became a race to add zeros. Everyone calls AI progress unpredictable. It's one of the most predictable things in technology. Labs fit this line on a ladder of small, cheap models first, read off where the giant one will land, and only then sign the check. They aren't gambling. They're reading a graph. Checkpoint 02 The $100M Bet Written on a Graph If loss is a straight line in compute, the next question is what compute actually costs - in chips and dollars. The answer is one almost insultingly simple equation: Six, times parameters, times training tokens. The six isn't a fudge factor: two FLOPs per parameter for the forward pass, plus about four for the backward pass. That's the whole cost model of the industry. Plug in GPT-3 - 175 billion parameters, 300 billion tokens - and you get about 3.15 × 10²³ operations, matching the public estimate almost to the digit. That number is the physical meaning of "GPT-3." Not a mind - a quantity of arithmetic. And it's why Nvidia, which sells the chip that does these exact multiplications, became one of the most valuable companies on Earth. The pace is worth feeling. Epoch AI found that since 2010, training compute has doubled roughly every six months - four to five times more every year, for over a decade. Moore's law, which defined computing for fifty years, doubled every two years. AI training compute has been doubling four times faster than that, without stopping. Nothing else in technology grows like this. And here's the detail I love. Look at the authors on that 2020 paper: Tom Brown, Dario Amodei, Alec Radford, Jared Kaplan. Four months later, most of that team shipped GPT-3 - a 175-billion-parameter bet placed on the exact curve they'd just drawn. A couple of years after that, many of them left to found Anthropic. The people who discovered the ruler built two of the biggest companies in AI by trusting it with billions. Everyone thinks the AI arms race is about smarter ideas. At the frontier, it's overwhelmingly about buying more of this one equation: pick a point further down the line, multiply until 6ND hits your budget, collect a smarter model. Checkpoint 03 The Paper That Proved Everyone Was Doing It Wrong For two years the industry followed Kaplan's advice, and it had a bias baked in: spend most of your compute on more parameters. Make the brain bigger. Microsoft and NVIDIA's MT-NLG hit a monstrous 530 billion. The parameter count became a leaderboard, and everyone climbed it. Then, in March 2022, DeepMind published Training Compute-Optimal Large Language Models and detonated the whole strategy. They trained over four hundred models to map where the loss actually bottoms out - and found everyone had been building models too big and feeding them too little. The finding - now called Chinchilla - is that model size and data should scale equally, both as the square root of compute: N {\text{opt}} \propto C^{0.5}, \qquad D {\text{opt}} \propto C^{0.5}, \qquad D \approx 20 \times N About twenty tokens per parameter. Now look at what the giants were actually doing: Every pre-Chinchilla giant was starved of data by a factor of ten. Enormous brains that had barely read anything. And DeepMind proved it in the most humiliating way possible: Chinchilla, at just 70 billion parameters trained on 1.4 trillion tokens, beat Gopher - a model four times its size - on the same compute. It beat GPT-3 and the 530-billion MT-NLG too. You can watch the waste by running GPT-3's own budget through the rule: The compute-optimal model at GPT-3's budget was around fifty billion parameters on a trillion tokens - smaller and better read than what OpenAI shipped. The most famous model of its era was leaving performance on the table by design. There's one honest catch. Chinchilla optimizes only the training bill, not the cost of serving the model to billions of users - where the parameter count is what you pay for, forever. So the pendulum swung again. Meta deliberately over-trained smaller models - Llama 3 on fifteen trillion tokens - because a small model is cheap to run for years. Chinchilla is the reason the era of small, over-read, laptop-sized models exists at all. Everyone was racing to build a bigger brain. The breakthrough was realizing they'd been starving it. More books beat a bigger head - and one paper reset the industry's blueprint in an afternoon. Checkpoint 04 Does Intelligence Really "Switch On"? Loss falls on a clean line. But loss is an abstract number in bits - what people care about is abilities. And that's where the clean story cracks into one of the best fights in AI. In 2022, a Google-led team including Jason Wei published Emergent Abilities of Large Language Models, and the claim was electric. Certain skills, they argued, are absent in small models and appear abruptly past a scale threshold - near-random, near-random, then a sudden snap into competence. They borrowed the physics word and called it a phase transition. Intelligence switches on like a light, at a size you can't predict. Then in 2023, a Stanford team - Schaeffer, Miranda, and Koyejo - fired back with Are Emergent Abilities of Large Language Models a Mirage?, and won one of NeurIPS 2023's top paper awards. Their argument is devastatingly simple: the jump is often an artifact of the yardstick, not the model. Grade a task all-or-nothing - every token right or you score zero - and per-token skill can rise smoothly the whole time while the score stays pinned at zero, until enough tokens line up and it appears to explode. You can make the mirage appear and vanish in a few lines: The per-token skill climbs like a ramp. The all-or-nothing score sits flat, then leaps - same model, same climb, no phase transition anywhere. The cliff lives in the scorecard. So who's right? Both, about different things. The underlying loss scales smoothly and predictably. Whether a human-meaningful ability appears to jump depends on how harshly you grade it. Everyone wants AI to have a magic threshold where it suddenly gets smart. The math says the smartness was rising smoothly the whole time. What looked like a light switching on was a continuous climb crossing the line where our scorecard finally gave it credit. Checkpoint 05 The Bitter Lesson Step back from any single paper and there's a deeper pattern underneath all of it - named years before the scaling era peaked. In 2019, Richard Sutton - a Turing Award winner and a founder of reinforcement learning - wrote a short essay called The Bitter Lesson. It's maybe the most-quoted thing in modern AI, and it's built to sting. His claim, from seventy years of history: general methods that leverage more computation always, eventually, win - by a large margin - over clever approaches that build in human knowledge. long-run performance⟶decided by compute, not human cleverness Chess engines full of hand-coded strategy lost to search. Hand-crafted vision features lost to deep nets that learned their own. Linguists' rules lost to models that just read the internet. Every time, the human-knowledge approach jumped ahead early - and got flattened once the compute caught up. And that compute keeps arriving on schedule: Roughly four billion times more compute than 2010. Against a tsunami like that, most clever tricks are sandcastles. The lesson is "bitter" because it's humbling - it says most of our hard-won insight is scaffolding that raw scale kicks away. It's an essay, not a proof, and people like Rodney Brooks and Gary Marcus push back. But for a decade, the scoreboard sided with Sutton. Everyone wants their clever trick to be the thing that matters. The most expensive lesson in AI is that, over and over, the clever trick got flattened by someone who just added compute and waited. Checkpoint 06 2024: The Year the Line Bent For a decade the recipe was one word - bigger - and it worked so reliably it became an article of faith. Then, in late 2024, the faith cracked. Through November 2024, The Information, Reuters, and Bloomberg all reported the same thing from inside the labs: the next generation at OpenAI, Google, and Anthropic showed a smaller jump than GPT-3 to GPT-4. Treat it as contested reporting - Sam Altman tweeted "there is no wall." But something real sat underneath it. We're running out of text. Epoch AI projects the entire stock of high-quality public human writing - around 300 trillion tokens - will be used up between 2026 and 2032. At NeurIPS 2024, Ilya Sutskever called data "the fossil fuel of AI" - finite, and being burned. You can't keep scaling D in 6ND when D is drying up. So the labs found a second dial: spend compute not at training time, but at inference time. Let the model think longer. OpenAI's o1 September 2024 showed accuracy rising with the logarithm of thinking compute: o3, announced that December, drove it home on the ARC-AGI benchmark, climbing to 87.5% - but its high-compute run spent thousands of dollars per task to get there. The new dial isn't free. A month later, in January 2025, China's DeepSeek released R1, an open-weights reasoning model that reproduced the behavior in public. And DeepSeek delivered the second shock, from the opposite direction. Their V3 reached near-frontier quality on a final training run reported at just a few million dollars. Read that carefully - it's the final run only, and excludes the hundreds of millions an all-in program actually costs. Most headlines destroyed that nuance. But the market didn't wait for nuance. On January 27, 2025, Nvidia fell about 17% in a single day, erasing close to $600 billion - the largest one-day loss in U.S. market history. The stock recovered, and the panic was partly a misread, but the message landed. Everyone thought scaling would either continue forever or hit a wall and stop. It did neither. Scaling didn't die - it forked. One dial became three: scale training, scale thinking-time, scale efficiency. The line didn't break. It grew branches. The Whole Story in One Breath Strip away the drama and the shape is clean. A 2020 OpenAI paper found AI error falls on a ruler-straight line as you add compute - predictable across seven orders of magnitude. The cost of moving down that line is one equation, six times parameters times tokens, doubled every six months for fifteen years. In 2022, DeepMind proved everyone had been starving their models of data, and reset the blueprint. The abilities that seemed to appear by magic were a smooth climb graded on a jagged scorecard. And in 2024, as the world's text ran dry, the line forked into three. The Line You Can't Uncross Seeing the line doesn't tell you whether it reaches all the way to real intelligence, and researchers disagree loudly about where it bends. But it drains the mysticism out of the room. The next time a lab unveils a model and the audience gasps, you'll know that a year earlier, someone fit a curve on a ladder of small runs, read off where this one would land, and signed a nine-figure check because the graph told them to. Most people will keep experiencing every release as a bolt from the blue, because they never saw the ruler under the magic. You just did - and now you can read where the same line points next. So here's the question I'll leave you with: if the smartest systems we've ever built have been sliding along a line we could see the whole time, how much of what's coming is actually a surprise - and how much is already sitting on the graph, drawn years ago, waiting for someone to buy the compute to reach it? // They call every new model a miracle. It was always a line on a graph. @ yusufknl reads the graph, weekly //