← The Gradient The number falling down my screen while a very small model trained is measured in bits. The same quantity, arrived at a century earlier from steam engines, says that destroying one of those bits must release heat.
To find out what the words in the machine learning papers actually point at, I built a language model small enough to watch. Fifty words of vocabulary. Ten dimensions per word. One self-attention layer with the usual query, key and value matrices, causally masked, then a feed-forward block, with the output weights tied back to the same embedding table. Sixteen hundred numbers in total, which is not a figure of speech: I can print all of them. It trains in a browser tab, one sentence per click, on a corpus of fifty sentences about animals sitting on things. What I could not stop watching was the loss, reprinted at every step, falling fast and then slowly and then, as we will see, not at all. It took me embarrassingly long to notice that this number is not a score. It is a physical quantity, it has units, and the units are bits.
The number is a count of questions #
Here is what the model is asked to do, stripped of everything else. Look at the sentence so far and put a probability on every word that could come next. Not a guess, a distribution: this much on dog, this much on ran, a sliver on quietly. Then the true next word arrives and the model is charged the logarithm of the probability it had assigned to that one, negated so the charge is positive. Confident and right is nearly free. Confident and wrong is ruinous. Spreading belief thinly over everything is a steady mediocre tax you pay forever.
<sub>x</sub>p(x) log
<sub>2</sub>q(x)
p is what the world does, q is what the model expects. In base 2 the answer is in bits. Training code almost always reports it in nats, using the natural logarithm; divide by ln 2 ≈ 0.693 to convert.
The base matters because base 2 makes the quantity picturable. Imagine you must identify the next word by asking yes or no questions, and you may design the questions as cleverly as you like given what you believe. The entropy is the average number of questions you need. That is not an analogy dressed as a definition; it is what Shannon proved in 1948, when he asked what function could measure uncertainty while satisfying a few obvious requirements and found that, up to a choice of units, only one function would do.
There is a second convention, perplexity, which is simply the entropy exponentiated, and which answers the question in the form people find intuitive: how large is the crowd the model is effectively choosing among. For my toy the arithmetic is unusually clean, because the vocabulary is exactly fifty words. A model with no opinion whatsoever spreads its probability evenly and pays ln 50, which is 3.91 nats, or 5.64 bits, a perplexity of exactly fifty. That is the number to beat, and it is the number you get for free by knowing nothing.
Before training, my model does slightly worse than knowing nothing: 4.14 nats, a perplexity of 63. Random initialization does not produce an even distribution, it produces a confidently arbitrary one, and confidence without cause is charged at the full rate. Three hundred training steps later, which is thirty seconds with the auto-train running, the same corpus costs 2.10 nats, or 3.03 bits per word, a perplexity of 8.2. The model went from choosing among sixty three words to choosing among eight. Nobody told it that the is usually followed by an animal. It was charged for not knowing, and it stopped not knowing.
Something else happened in those same three hundred steps, which I checked afterwards because the previous essay in this series had claimed it. At initialization the eight animal words point in unrelated directions in the ten dimensional space, average cosine similarity 0.05, which is what random vectors do. By step 300 that average is 0.48, while words from different categories drift to slightly negative similarity. The animals were never labelled as animals. They became neighbours because they kept being charged for the same mistakes.
Shannon put people in a room with a book #
A perplexity of 8.2 is only interesting if you know what perfect would have been, and this is exactly the question Shannon faced in 1951 for English. He wanted the entropy of the language itself, and he could not compute it, because nobody possesses the true distribution over English sentences. So he did something that reads today like performance art. He sat a person down with a passage from a book they had not read, revealed it one letter at a time, and made them guess the next letter, repeating until they got it right and recording how many guesses it took. A person who knows English is a language model. He simply read the model out of the person.
Reduce English to 27 symbols, twenty six letters and a space, and the milestones fall in a line. Guess uniformly and you need log<sub>2</sub> 27, about 4.76 bits per letter. Use nothing but the frequency of individual letters, the fact that e is common and z is not, and you reach roughly 4.03. Use pairs, then triples, and you descend into the low threes. Shannon's human subjects, given a hundred characters of context, landed between 0.6 and 1.3 bits per character. The upper end of that range is the number usually quoted, and it is worth saying what it claims: a fluent reader, at each letter, is choosing between about two live possibilities.
I cannot put my model on that chart, and the reason is the most useful thing in this essay. Shannon had to measure the floor because English is bigger than any description of it. My toy language is not. It is fifty sentences, 297 words, and I own all of it, which means the floor is not an estimate but a calculation. Take the corpus as the entire world, ask at every position what a perfect memorizer would predict given the exact words so far, and average the surprise that remains. The answer is 0.67 nats, 0.96 bits per word, a perplexity of 1.95. It is not zero, because sentences share openings and a model that has seen everything still cannot know whether the dog ran or slept. That residue is the irreducible uncertainty of a world made of fifty sentences, and no model, however large, may go below it.
That warning is not pedantry, it is the most commonly broken rule in the field. Two loss curves from two tokenizers cannot be compared at all, because a model charged once for a five letter chunk and a model charged once per letter are being billed on different plans. Bits per character is the honest common denominator for text precisely because the tokenizer cancels out of it. Bits per token does not, and both numbers print in the same terminal in the same font.
What survives the caveat is the shape. There is a floor under this game, set by how much of the world is really unpredictable, and every model ever trained sits somewhere on the ladder above it. Mine got about 56 percent of the way down in thirty seconds, and 88 percent of the way down if you leave the auto-train running for about eight minutes. Training a language model is not, at bottom, teaching a machine to write. It is squeezing a corpus, and the loss curve is a compression ratio being negotiated in public.
I had met the formula before, in another notation #
When I first wrote out the cross entropy by hand I had the distinct feeling of having seen it somewhere else, and it took a day to place it. It was on the bench, years earlier, in a thermodynamics course, wearing different letters.
Mix two liquids that do not interact, and the mixing happens for no energetic reason at all. Nothing is released, no bond is improved. It happens because there are overwhelmingly more arrangements of the molecules in which the two are intermingled than arrangements in which they are apart, and the system spends its time in the arrangements there are more of. The quantity that counts those arrangements, for an ideal mixture, is this:
<sub>mix</sub>= −nR ∑
<sub>i</sub>x
<sub>i</sub>ln x
<sub>i</sub>against H = −∑
<sub>i</sub>p
<sub>i</sub>log
<sub>2</sub>p
<sub>i</sub>
the entropy of mixing, from the bench, and the entropy of a message, from the telephone exchange. The mole fractions of a solution are a probability distribution. The two expressions differ by a physical constant and a choice of logarithm base.
These are not cousins. They are the same function. The mole fractions of my formulation sum to one and are non negative, which is to say they are a probability distribution, and if you feed them to Shannon's formula you get the entropy of mixing back, divided by nR. A chemist calculating whether a blend will stay blended and an engineer calculating whether a line can carry a signal are evaluating one expression, and neither field borrowed it from the other. Gibbs wrote it in 1878 for statistical mechanics. Shannon arrived at it in 1948 from a list of properties a measure of uncertainty ought to have, and only afterwards was told what it already had a name.
The story of how it got the name is too good to leave out, and I will report it as the anecdote it is. Shannon, according to an account Myron Tribus gave years later, asked von Neumann what to call his quantity. Call it entropy, von Neumann is supposed to have said, for two reasons: the same function already exists in statistical mechanics, and more importantly, nobody really understands entropy, so in a debate you will always have the advantage.
The joke has teeth. Entropy is taught to chemists as disorder, which is a metaphor that survives contact with a stirred beaker and dies immediately on contact with anything else. It is not disorder. It is a count of how many microscopic arrangements are compatible with what you know macroscopically, which makes it a statement about the relationship between a description and a system, not a property sitting inside the system waiting to be measured. That reading was pushed hardest by Edwin Jaynes in 1957, who argued that statistical mechanics is not really a physical theory at all but an exercise in inference: given the little you know, adopt the distribution that assumes the least beyond it, which is the one of maximum entropy. On that view thermodynamics was doing information theory for eighty years without the vocabulary.
Erasing a bit warms the room #
If entropy in a message and entropy in a beaker are the same function, that is worth taking seriously, not just noting. Thermodynamic entropy carries a unit with a joule in it. Information entropy is just a count of questions, no units at all. Either the resemblance is a coincidence, the kind where one equation happens to describe both a thrown ball and a suspension cable without the two having anything to do with each other, or there is an actual exchange rate between the two: a number of joules for every bit. The question is old and it starts with a joke that would not die. In 1867 Maxwell imagined a tiny being at a shutter between two boxes of gas, letting fast molecules through one way and slow ones the other, sorting hot from cold and doing no work. The second law of thermodynamics, apparently, defeated by attention. For sixty years the demon embarrassed physics. Szilard got closer in 1929 by shrinking the whole apparatus to a single molecule and showing that the demon's act of finding out which side it was on was worth exactly kT ln 2 of work, but the accounting still had a hole in it.
Rolf Landauer closed it in 1961, and the answer is stranger than the question. Measurement is not what costs. What costs is forgetting. Any operation you cannot run backward, one where the output no longer tells you what the input was, throws information away, and throwing information away is what costs energy. Erase one bit, take a memory cell that could have been in either of two states and force it into one, and you must dump at least kT ln 2 of energy into the surroundings as heat. At room temperature that is about three zeptojoules, which is three thousandths of a billionth of a billionth of a joule.
<sub>min</sub>= k
<sub>B</sub>T ln 2 ≈ 2.9 × 10
<sup>−21</sup>J at 300 K
the price of destroying one bit. Not the price of computing with it, not the price of reading it: the price of no longer being able to say what it was.
Charles Bennett drew the moral in 1982 and it finally killed the demon. Computation itself can in principle be done reversibly, at a cost approaching zero, so long as you keep everything. But the demon has a notebook, and the notebook is finite. Sooner or later it has to be cleared to keep sorting, and at that moment the bill arrives, exactly large enough to save the second law. The demon was never getting something for nothing. It was running a tab.
This all sounds like the kind of argument that stays on a blackboard, so the part I like best is that it does not. In 2012 a group in Lyon built the experiment: a single micron-sized bead held in a double-well optical trap, one well for zero and one for one, a real physical bit. They erased it, over and over, and measured the heat that came out. The dissipation converged on kT ln 2 from above, as predicted, in a real laboratory, with error bars. Information is not a metaphor drawn from physics. It is physics.
Which brings me back to my browser tab. I instrumented the toy and counted the arithmetic exactly: 30,632 multiply-accumulates per training step, 9.19 million of them for the three hundred steps that took the perplexity from 63 to 8.2. Suppose, generously, that every one of those operations destroys a full 64 bit intermediate result. Landauer's bill for the whole training run is then about 1.7 picojoules. The laptop, running a JavaScript engine and a screen for the thirty seconds it took, spent something like six hundred joules. The ratio is around three hundred trillion.
I want to be careful about what that gap is. Almost none of it is the arithmetic. It is the screen, the operating system, the browser, the transistors switching far above threshold for reliability, all the engineering that makes a computer usable rather than optimal. The thermodynamic cost of what my model actually learned would not warm a room, or a fingertip, or anything measurable. But there is a floor down there, it is made of nothing but temperature, and it is the only line in this business that no better engineer will ever move.
What the bill does not buy #
There is a tempting extra step here, and it is worth resisting. A model that reaches a low cross entropy on English has found real structure in English, and it is fair to call that compression. Whether it is fair to call it understanding is a bigger claim, one a standing prize for compressing Wikipedia is built on betting yes to. Cross entropy alone cannot settle it: the quantity measures agreement with whatever distribution the model was shown, not agreement with the world. Train on confident nonsense and a model that reaches one bit per character becomes an excellent model of confident nonsense, and its loss curve looks exactly as satisfying either way. This is the whole reason a fluent model can be wrong with a straight face. Nothing in the objective ever mentioned truth, only how surprised to be.
My own toy made the same point on a smaller scale. Left running too long, the first version of it blew up: the weights ran away, the probabilities underflowed, and the loss pinned at 27.631 for a few steps before going to NaN. That number was never a measurement of the model. It was the guard rail the code adds to avoid dividing by zero, printing a confident-looking number long after it had stopped meaning anything.
I had met this name before, too #
Two classes from my own electrical engineering degree turn out to matter here too. Digital logic was built entirely out of Shannon: his 1937 MIT thesis showed that Boolean algebra, true and false, and, or, not, could be wired up out of ordinary relay switches. Before that thesis, circuit design was trial and error. After it, circuit design was a proof, which is the reason a multiply-accumulate in any computer, including the toy in this essay, can be built out of transistors at all. A Symbolic Analysis of Relay and Switching Circuits is sometimes called the most important master's thesis of the twentieth century.
The other class, signals and systems, left one more number everyone carries for life: sample a signal at twice its highest frequency or you cannot get it back. That is the Nyquist-Shannon sampling theorem, and Shannon gave it its rigorous proof in a 1949 paper. The same math shows up, unannounced, inside the Transformer. Self-attention has no sense of word order on its own; shuffle a sentence and it returns the same answer. The fix in the original paper is to add a wave built from sine and cosine at a ladder of frequencies to every position, so order gets encoded the way a signal gets encoded: as frequency. Attention Is All You Need, section 3.5, is a page of signal-processing math doing a job nobody in signal processing designed it for.
FormLab's own mark has a wavy line across the bottom of it, meant to read as liquid in a flask. After a few weeks staring at sine waves for this essay, it is hard to look at that line now and not see one, though I doubt anyone drew it thinking about Fourier analysis. FormLab is formulation software, not a signal-processing tool, and the line was meant to be liquid. One connection I did not have to go looking for: I wrote this essay talking to an AI named Claude, and Anthropic has said the name is an homage to Shannon: the switches, the sine waves, and now the byline, all one man.
Coda #
What I did not expect from a few weeks of training toy models was to end up back in a thermodynamics lecture. The number falling down my screen is a count of questions. The same expression, with a physical constant in front of it, tells me whether the two liquids on my bench will stay mixed. And the connection between the two is not a pun, because there is an exchange rate, and someone has measured it with a bead in a laser trap.
I am wary of the grand version of this thought, the one where everything is information and the universe is a computer. The honest statement is smaller and I think better. There is a quantity that shows up whenever a description falls short of a system, and it does not care whether the shortfall is a chemist's, a telephone engineer's, or a neural network's. It is measured in the same units every time. And if you insist on destroying some of it, the room gets warmer, by an amount you can look up.
Notes & further reading
- A Mathematical Theory of Communication - Shannon (1948). The founding paper, and still the clearest statement of why the entropy formula is the only one that could have worked.
- Prediction and Entropy of Printed English - Shannon (1951). The guessing experiment, the tables behind the ladder in the figure, and the bounds of 0.6 to 1.3 bits per character.
- A Convergent Gambling Estimate of the Entropy of English - Cover & King (1978). The same measurement made sharper by having subjects bet rather than guess, landing near 1.25 bits per character.
- On the decrease of entropy in a thermodynamic system by the intervention of intelligent beings - Szilard (1929). One molecule, one bit, and the first appearance of kT ln 2 as a price.
- Irreversibility and Heat Generation in the Computing Process - Landauer (1961). Four pages that move the cost from measuring to forgetting.
- The Thermodynamics of Computation - a Review - Bennett (1982). Reversible computation, and the demon's notebook as the place the bill finally lands.
- Experimental verification of Landauer's principle - Bérut et al., Nature (2012). A colloidal bead in a double-well trap, erased repeatedly, with the heat measured.
- Information Theory and Statistical Mechanics - Jaynes (1957). The argument that thermodynamics is inference, and that maximum entropy is what honesty looks like in equations.
- Energy and Information - Tribus & McIrvine, Scientific American (1971). The source usually given for the von Neumann naming story, which should be enjoyed as an anecdote rather than cited as a minute.
- A Symbolic Analysis of Relay and Switching Circuits - Shannon (1937, published 1938). The thesis that turned circuit design into algebra, and the reason any of the arithmetic in this essay can be built out of switches at all.
- Communication in the Presence of Noise - Shannon, Proc. IRE (1949). The rigorous sampling theorem, and the channel capacity derived from it.
- Attention Is All You Need - Vaswani et al. (2017). Section 3.5 is the sinusoidal positional encoding; the rest of the paper is the architecture nearly everything since has copied.
- Ever tried Claude.AI? Meet the Michigander who inspired its name - Bridge Michigan (2026). The source for the Anthropic naming claim above.
- The Hutter Prize . A standing bounty for compressing a fixed extract of Wikipedia, run on the explicit thesis that better compression is better understanding. Worth arguing with in either direction.
The Gradient is a series of essays on the places where science and mathematics meet. It is not about FormLab, and it is not trying to sell you anything. New pieces land irregularly, when there is something worth saying.
Written by Jun Liu. Follow along by RSS (no email, no account - your reader, your terms), or read the rest of The Gradient →