Before GPUs. Before neural networks. Before transistors were even a rumor.
There was a hungry cat in a wooden crate, trying to get to a piece of fish.
The main character of this era isn't an engineer. It's a loose group of psychologists — some of whom actively disliked each other's ideas — poking at animals to answer one question: why do living things do the things they do?
They never wrote a line of code. They were not trying to build AI. And yet, in the fifty years between 1898 and 1949, they collectively reverse-engineered the core loop that every modern reinforcement learning system still runs on.
This post is that story. No math yet — I promise. Just the loop, and the four people who found its pieces.
Let me give you the shape of the answer before the story, so you know what you're collecting. Reinforcement learning describes any system that learns by doing stuff and seeing what happens. That's it. Not by being shown correct answers — by trying, failing, and adjusting.
We'll call the learner the agent (the cat, the rat, the robot, the model). Everything outside it is the environment (the box, the room, the game). The loop runs like this:
Two more, and they're the ones that do the heavy lifting later.
The agent needs a policy: a strategy for choosing actions. Not a fixed script — a rule of thumb that says in this kind of situation, do this kind of thing.
And it needs a sense of value: how good a state is, based not on the reward sitting in it right now, but on everything it tends to lead to. An empty hallway is worth nothing on its own. An empty hallway with the kitchen at the end of it is worth quite a lot.
Six words: agent, state, action, reward, policy, value. Every single one of them was discovered by someone studying animals, decades before anyone could program a computer.
Let's go get them.
In 1898, a graduate student named Edward Thorndike was annoyed.
The animal literature of his day was full of charming anecdotes — dogs who understood latches, cats who figured out doorknobs. Thorndike's suspicion was that people were flattering their pets. Animals weren't reasoning. They were doing something else, and nobody had bothered to measure it.
So he built the puzzle box. The setup is almost comically simple. A hungry cat goes in a small wooden crate. The door opens only if the cat pulls a loop of string, or presses a lever, or steps on a pedal. A piece of fish sits outside, in plain view. Then Thorndike started a stopwatch.
Trial one is chaos. The cat claws the bars. It shoves its paws through every gap. It bites things, yowls, thrashes at anything loose. After a few minutes of pure flailing, it hits the lever by accident. The door swings open. Fish.
Trial two: slightly less flailing.
Trial twenty: the cat walks in, hits the lever, and strolls out.
Here's what Thorndike noticed, and it's the part that matters: there was never a lightbulb moment. No trial where the cat suddenly got it. If cats were reasoning about levers, you'd expect the escape time to fall off a cliff — one long trial, then instantly fast ones. Instead, when he plotted escape time against trial number, he got a slow, noisy, gradual curve. The useless behaviors quietly faded. The useful one got faster.
That curve is the first real evidence in this story. Learning wasn't insight. It was selection — good behaviors surviving, bad ones dying off.
Thorndike wrote it up as the Law of Effect:
If an action is followed by a satisfying state of affairs, the bond between that situation and that action is strengthened. If followed by discomfort, it is weakened. Read that again with our vocabulary. Situation is the state. Action is the action. Satisfying is the reward. Bond strengthened is the policy update.
Thorndike, in 1898, wrote pseudocode for a policy gradient. He just used the word "bond."
🎒
In the backpack:trial-and-error learning. Rewards make actions more likely. Learning is gradual, not sudden.
Thorndike's cat only learns after it gets the fish. The reward does all the teaching.
But watch a real animal for five minutes and you'll notice it reacting to things that aren't rewards at all. Your dog loses its mind at the sound of a cupboard opening. Nothing has been eaten yet. The cupboard predicts food.
Thorndike had no account of that. Rewards were where his story started and ended.
The person who noticed the signal was, characteristically, not looking for it.
Ivan Pavlov was a Russian physiologist studying digestion. He won a Nobel Prize for it in 1904. He was, by most accounts, mildly irritated that history remembers him for the dogs.
His actual research required measuring saliva. And his dogs kept ruining the measurements by salivating early — at the sight of the lab assistant who usually brought food, at the sound of footsteps in the corridor. Before any food existed.
Most researchers would call that noise and try to control for it. Pavlov called it the experiment.
He ran it systematically. Present a neutral signal — a metronome, a buzzer, a light, a tone — then deliver food, a second or two later. Repeat.
(Quick myth-correction, since it costs one sentence: the famous bell is mostly folklore. Pavlov's lab notes lean heavily on metronomes, buzzers, and tuning forks. The bell survived because it's easier to draw.)
After enough pairings, the signal alone made the dogs salivate. Nothing about the food had changed. What changed was that the dog had learned the signal carries information about the future.
This is the second piece, and it's a big one. Thorndike's cat learned what to do. Pavlov's dog learned what's coming.
In RL terms, that's value. The value of a state is roughly: how much good stuff do I expect from here on out? A metronome that reliably precedes food is a high-value state — not because a metronome is nutritious, but because of what tends to follow it.
You do this constantly. The smell of coffee is not caffeine. Your inbox badge is not the email. A green light is not arriving anywhere. Every one of these is a state you've assigned value to, purely because of what historically came next.
And this is why RL agents can learn anything at all in sparse environments. A chess engine that only felt reward at checkmate would learn approximately nothing — the signal is 60 moves away from most of the decisions that caused it. What it actually learns is to value positions: this pawn structure tends to precede winning. That's a metronome.
🎒
In the backpack:states have value. Value comes fromprediction, not from the reward itself.
Pavlov's dogs were passive. They sat in a harness and predicted. The dog's salivation doesn't make the food arrive any sooner.
So now we have two halves of an agent and no glue:
| Thorndike's cat | Pavlov's dog | |
|---|---|---|
| Learns | ||
| which actions pay off | which signals predict payoff | |
| Can it act? | ||
| ✅ | ❌ | |
| Can it anticipate? | ||
| ❌ | ✅ |
A real agent needs both — and it needs someone to make this a controllable science rather than a collection of observations. That took a genuine control freak.
B.F. Skinner's question was different from everyone before him. Not "how do animals learn?" but: "how precisely can I control what an animal learns?"
His tool was the operant conditioning chamber — the Skinner Box. A rat inside can press a bar. A pigeon can peck a disc. Food is dispensed on rules Skinner sets. Simple inputs, measurable outputs, and — critically — the animal stays in the box.
That last detail seems minor and isn't. Thorndike's cats escaped, which meant one trial, then a reset by hand. Skinner's animals could run thousands of trials in an afternoon, uninterrupted. He'd built the first high-throughput training environment. Every RL researcher who has ever run a simulator overnight is doing the same thing.
Two of his findings matter for us.
Skinner varied when reward arrived — every press, every fifth press, every fifth press on average. These are schedules of reinforcement.
The counterintuitive result: variable reward — where the animal can't predict which press pays — produced behavior that was hardest to extinguish. Reward a pigeon every single time, then stop, and it gives up quickly. Reward it unpredictably, then stop, and it keeps going for a remarkably long time.
Slot machines run on this. So do infinite-scroll feeds. Skinner found the exploit in the 1950s; the attention economy just productized it.
For us, the useful version is that reward structure is a design decision with enormous consequences. Which sets up an entire genre of modern RL failure — agents that learn the reward signal instead of the task, in ways nobody intended. We'll spend real time there in a later post. The bigger idea is shaping.
Skinner taught pigeons to play a passable game of table tennis. Not by waiting for a pigeon to spontaneously play table tennis — you'd wait forever. He rewarded incremental steps. First, reward the bird for merely facing the table. Once that's reliable, stop rewarding it and only pay for approaching the ball. Then only for contact. Then only for direction.
https://www.youtube.com/shorts/RPJyKSmH9WM?feature=share You are moving the goalposts on purpose, in small enough increments that the animal never falls off the curve.
This is reward shaping, and it's how we train robots today. A robot arm rewarded only for "successfully assembled the part" will flail randomly forever, because the odds of stumbling into success by accident are effectively zero. So we hand it crumbs: reward for moving toward the part, then for gripping it, then for lifting. Same ladder. Different species.
And the strategy the animal ends up with — in this situation, do this — is exactly what we called a policy at the top of this post. Skinner spent thirty years training policies. He just did it with pigeons and a food dispenser instead of gradients and a GPU.
🎒
In the backpack:policies can betrained, not just observed. Reward timing and reward structure change everything.
We now have a full behavioral picture: act, predict, get shaped, improve.
Every bit of it is described from the outside. Skinner was deliberate about this — he treated the mind as a black box and argued you should only ever talk about observable behavior. Which is scientifically disciplined and completely useless if you want to build one.
Because if you're going to build this thing, you need to answer a question Skinner refused to ask: where does the learning physically live?
Donald Hebb, a Canadian psychologist, published The Organization of Behavior in 1949 and proposed a mechanism so simple it fits in a sentence.
If neuron A repeatedly takes part in firing neuron B, some growth process or metabolic change takes place in both cells such that A's efficiency in firing B is increased. That's Hebb's Rule, and the pop version is stickier: neurons that fire together, wire together.
Two neurons active at the same time? The connection between them — the synapse — physically strengthens. Next time, A triggers B more easily. Do it enough and the path becomes a highway.
Run Thorndike's cat through this. The cat sees the lever (a pattern of neurons firing) and presses it (another pattern firing). They're active together. Their connection thickens. A hundred trials later, seeing the lever is pressing the lever — there's a paved road between them.
That's a habit, described in hardware.
And this is where artificial neural networks come from. In a network, the connection strength between two artificial neurons is a number called a weight. Learning means adjusting weights so that useful inputs reliably produce useful outputs. Hebb is the reason "learning = changing connection strengths" is the default assumption in the entire field.
One important correction, because this gets flattened a lot: Hebb's rule is not backpropagation, and the difference is worth thirty seconds.
Hebb's rule is local. A synapse strengthens based only on what its own two neurons are doing right now. No supervisor, no error signal, nobody keeping score.
Backpropagation — the algorithm that actually trains modern networks — is global and error-driven. It compares the network's output to a desired output, computes how wrong it was, and pushes that error signal backward through every layer, adjusting each weight by how much it contributed to the mistake. Every weight update depends on information from far away in the network.
These are genuinely different mechanisms. In fact, the gap between them is one of the liveliest open arguments in neuroscience: brains clearly don't run backprop — there's no known biological machinery for shipping a precise error signal backward across a network — yet backprop works spectacularly well. Whether the brain approximates something like it, or does something else entirely, is unresolved.
So the honest lineage is: Hebb gave us the idea that memory lives in connection strengths. Backprop is a much later, mathematically-driven answer to how to update those strengths. Same neighborhood, different building.
🎒
In the backpack:learning is stored in connection weights. Local, physical, incremental.
Fifty-one years, four researchers, and the loop is complete:
| Idea | From | What it becomes |
|---|---|---|
| Rewards strengthen the actions that earned them | Thorndike, 1898 | the reward signal |
| Signals predict future reward | Pavlov, 1900s | value |
| Behavior can be sculpted step by step | Skinner, 1930s–50s | policies, reward shaping |
| Learning is stored in connection strengths | Hebb, 1949 | weights |
Read that table top to bottom and you have a specification. Something that acts, predicts, gets shaped by consequences, and stores what it learned in the strength of its connections.
Nobody had built it. Nobody could — the ideas were all written in the language of biology, describing wet tissue that nobody could manufacture.
But 1949 is the year that stops being true. The war had just produced digital computers. A new field called cybernetics was arguing that control and feedback worked the same way in animals and machines, and that the distinction was less fundamental than everyone assumed.
For the first time, people had a spec for learning and a machine to run it on. Before we leave the psychologists, here's the crack in the foundation — worth holding onto, because it takes another twenty years to fix.
Every idea above says learning happens when things co-occur. Metronome and food together → learn. Lever and fish together → learn. Neurons active together → wire together.
But co-occurrence can't be the whole story:
Your alarm has gone off at 7am every day for a year. By the co-occurrence rule, this morning should have strengthened that association a little further, same as every other morning. It didn't. You learned it in the first week and then stopped — even though the pairing kept happening, perfectly, hundreds more times.
So what switches learning off?
Nobody in this post can answer that. The answer arrives in 1972, from a pair of psychologists still studying rats, and when it does it turns out to be the single most important equation in this entire series. We'll get there.
RL #2 — Cybernetics and Early Machines (1948–1954). The first people to actually build this thing. Alan Turing sketching a "pleasure-pain system" in 1948. Claude Shannon's Theseus, a mechanical mouse that learned a maze by trial and error and could remember it. And Marvin Minsky wiring together the SNARC — 40 vacuum tubes, motors, and a clutch, arguably the first machine that learned by reinforcement.
The cats gave us the algorithm. Now watch people try to solder it.
If you spot an error or think I've oversimplified something, drop it in the comments — I'd rather fix it than have it sit there.