{"slug": "rl-1-biological-foundations-and-the-law-of-effect-1898-1949", "title": "RL 1: Biological foundations and the \"Law of Effect\" (1898– 1949)", "summary": "A historical account traces the origins of reinforcement learning to animal psychology experiments between 1898 and 1949, highlighting Edward Thorndike's puzzle box studies and the formulation of the Law of Effect. The narrative explains how core RL concepts—agent, state, action, reward, policy, and value—were discovered by psychologists studying animal behavior, long before modern computing.", "body_md": "Before GPUs. Before neural networks. Before transistors were even a rumor.\n\nThere was a hungry cat in a wooden crate, trying to get to a piece of fish.\n\nThe main character of this era isn't an engineer. It's a loose group of psychologists — some of whom actively disliked each other's ideas — poking at animals to answer one question: *why do living things do the things they do?*\n\nThey never wrote a line of code. They were not trying to build AI. And yet, in the fifty years between 1898 and 1949, they collectively reverse-engineered the core loop that every modern reinforcement learning system still runs on.\n\nThis post is that story. No math yet — I promise. Just the loop, and the four people who found its pieces.\n\nLet me give you the shape of the answer before the story, so you know what you're collecting.\n\nReinforcement learning describes any system that learns by **doing stuff and seeing what happens.** That's it. Not by being shown correct answers — by trying, failing, and adjusting.\n\nWe'll call the learner the **agent** (the cat, the rat, the robot, the model). Everything outside it is the **environment** (the box, the room, the game). The loop runs like this:\n\nTwo more, and they're the ones that do the heavy lifting later.\n\nThe agent needs a **policy**: a strategy for choosing actions. Not a fixed script — a rule of thumb that says *in this kind of situation, do this kind of thing.*\n\nAnd it needs a sense of **value**: how good a state is, based not on the reward sitting in it right now, but on everything it tends to lead to. An empty hallway is worth nothing on its own. An empty hallway *with the kitchen at the end of it* is worth quite a lot.\n\nSix words: **agent, state, action, reward, policy, value.** Every single one of them was discovered by someone studying animals, decades before anyone could program a computer.\n\nLet's go get them.\n\nIn 1898, a graduate student named Edward Thorndike was annoyed.\n\nThe animal literature of his day was full of charming anecdotes — dogs who *understood* latches, cats who *figured out* doorknobs. Thorndike's suspicion was that people were flattering their pets. Animals weren't reasoning. They were doing something else, and nobody had bothered to measure it.\n\nSo he built the [puzzle box](https://www.appstate.edu/~steelekm/classes/psy5300/Documents/chance-72-433.pdf).\n\nThe setup is almost comically simple. A hungry cat goes in a small wooden crate. The door opens only if the cat pulls a loop of string, or presses a lever, or steps on a pedal. A piece of fish sits outside, in plain view. Then Thorndike started a stopwatch.\n\n**Trial one is chaos.** The cat claws the bars. It shoves its paws through every gap. It bites things, yowls, thrashes at anything loose. After a few minutes of pure flailing, it hits the lever *by accident.* The door swings open. Fish.\n\n**Trial two:** slightly less flailing.\n\n**Trial twenty:** the cat walks in, hits the lever, and strolls out.\n\nHere's what Thorndike noticed, and it's the part that matters: **there was never a lightbulb moment.** No trial where the cat suddenly *got it.* If cats were reasoning about levers, you'd expect the escape time to fall off a cliff — one long trial, then instantly fast ones. Instead, when he plotted escape time against trial number, he got a slow, noisy, gradual curve. The useless behaviors quietly faded. The useful one got faster.\n\nThat curve is the first real evidence in this story. Learning wasn't insight. It was **selection** — good behaviors surviving, bad ones dying off.\n\nThorndike wrote it up as the **Law of Effect**:\n\nIf an action is followed by a satisfying state of affairs, the bond between that situation and that action is strengthened. If followed by discomfort, it is weakened.\n\nRead that again with our vocabulary. *Situation* is the **state**. *Action* is the **action**. *Satisfying* is the **reward**. *Bond strengthened* is the **policy update**.\n\nThorndike, in 1898, wrote pseudocode for a policy gradient. He just used the word \"bond.\"\n\n🎒\n\nIn the backpack:trial-and-error learning. Rewards make actions more likely. Learning is gradual, not sudden.\n\nThorndike's cat only learns *after* it gets the fish. The reward does all the teaching.\n\nBut watch a real animal for five minutes and you'll notice it reacting to things that aren't rewards at all. Your dog loses its mind at the sound of a cupboard opening. Nothing has been eaten yet. The cupboard *predicts* food.\n\nThorndike had no account of that. Rewards were where his story started and ended.\n\nThe person who noticed the signal was, characteristically, not looking for it.\n\nIvan Pavlov was a Russian physiologist studying digestion. He won a Nobel Prize for it in 1904. He was, by most accounts, mildly irritated that history remembers him for the dogs.\n\nHis actual research required measuring saliva. And his dogs kept ruining the measurements by salivating **early** — at the sight of the lab assistant who usually brought food, at the sound of footsteps in the corridor. Before any food existed.\n\nMost researchers would call that noise and try to control for it. Pavlov called it the experiment.\n\nHe ran it systematically. Present a neutral signal — a metronome, a buzzer, a light, a tone — then deliver food, a second or two later. Repeat.\n\n*(Quick myth-correction, since it costs one sentence: the famous bell is mostly folklore. Pavlov's lab notes lean heavily on metronomes, buzzers, and tuning forks. The bell survived because it's easier to draw.)*\n\nAfter enough pairings, the signal alone made the dogs salivate. Nothing about the food had changed. What changed was that the dog had learned **the signal carries information about the future.**\n\nThis is the second piece, and it's a big one. Thorndike's cat learned *what to do.* Pavlov's dog learned *what's coming.*\n\nIn RL terms, that's **value**. The value of a state is roughly: *how much good stuff do I expect from here on out?* A metronome that reliably precedes food is a high-value state — not because a metronome is nutritious, but because of what tends to follow it.\n\nYou do this constantly. The smell of coffee is not caffeine. Your inbox badge is not the email. A green light is not arriving anywhere. Every one of these is a state you've assigned value to, purely because of what historically came next.\n\nAnd this is why RL agents can learn anything at all in sparse environments. A chess engine that only felt reward at checkmate would learn approximately nothing — the signal is 60 moves away from most of the decisions that caused it. What it actually learns is to value *positions*: this pawn structure tends to precede winning. That's a metronome.\n\n🎒\n\nIn the backpack:states have value. Value comes fromprediction, not from the reward itself.\n\nPavlov's dogs were passive. They sat in a harness and predicted. The dog's salivation doesn't make the food arrive any sooner.\n\nSo now we have two halves of an agent and no glue:\n\n| Thorndike's cat | Pavlov's dog | |\n|---|---|---|\nLearns |\nwhich actions pay off | which signals predict payoff |\nCan it act? |\n✅ | ❌ |\nCan it anticipate? |\n❌ | ✅ |\n\nA real agent needs both — and it needs someone to make this a controllable science rather than a collection of observations. That took a genuine control freak.\n\nB.F. Skinner's question was different from everyone before him. Not \"how do animals learn?\" but: **\"how precisely can I control what an animal learns?\"**\n\nHis tool was the operant conditioning chamber — the Skinner Box. A rat inside can press a bar. A pigeon can peck a disc. Food is dispensed on rules Skinner sets. Simple inputs, measurable outputs, and — critically — the animal *stays in the box.*\n\nThat last detail seems minor and isn't. Thorndike's cats escaped, which meant one trial, then a reset by hand. Skinner's animals could run thousands of trials in an afternoon, uninterrupted. He'd built the first high-throughput training environment. Every RL researcher who has ever run a simulator overnight is doing the same thing.\n\nTwo of his findings matter for us.\n\nSkinner varied *when* reward arrived — every press, every fifth press, every fifth press *on average.* These are **schedules of reinforcement.**\n\nThe counterintuitive result: **variable** reward — where the animal can't predict which press pays — produced behavior that was hardest to extinguish. Reward a pigeon every single time, then stop, and it gives up quickly. Reward it unpredictably, then stop, and it keeps going for a remarkably long time.\n\nSlot machines run on this. So do infinite-scroll feeds. Skinner found the exploit in the 1950s; the attention economy just productized it.\n\nFor us, the useful version is that reward *structure* is a design decision with enormous consequences. Which sets up an entire genre of modern RL failure — agents that learn the reward signal instead of the task, in ways nobody intended. We'll spend real time there in a later post.\n\nThe bigger idea is **shaping.**\n\nSkinner taught pigeons to play a passable game of table tennis. Not by waiting for a pigeon to spontaneously play table tennis — you'd wait forever. He rewarded *incremental steps.* First, reward the bird for merely facing the table. Once that's reliable, stop rewarding it and only pay for approaching the ball. Then only for contact. Then only for direction.\n\n[https://www.youtube.com/shorts/RPJyKSmH9WM?feature=share](https://www.youtube.com/shorts/RPJyKSmH9WM?feature=share)\n\nYou are moving the goalposts on purpose, in small enough increments that the animal never falls off the curve.\n\nThis is **reward shaping**, and it's how we train robots today. A robot arm rewarded only for \"successfully assembled the part\" will flail randomly forever, because the odds of stumbling into success by accident are effectively zero. So we hand it crumbs: reward for moving toward the part, then for gripping it, then for lifting. Same ladder. Different species.\n\nAnd the strategy the animal ends up with — *in this situation, do this* — is exactly what we called a **policy** at the top of this post. Skinner spent thirty years training policies. He just did it with pigeons and a food dispenser instead of gradients and a GPU.\n\n🎒\n\nIn the backpack:policies can betrained, not just observed. Reward timing and reward structure change everything.\n\nWe now have a full behavioral picture: act, predict, get shaped, improve.\n\nEvery bit of it is described from the outside. Skinner was deliberate about this — he treated the mind as a black box and argued you should only ever talk about observable behavior. Which is scientifically disciplined and completely useless if you want to *build* one.\n\nBecause if you're going to build this thing, you need to answer a question Skinner refused to ask: **where does the learning physically live?**\n\nDonald Hebb, a Canadian psychologist, published *The Organization of Behavior* in 1949 and proposed a mechanism so simple it fits in a sentence.\n\nIf neuron A repeatedly takes part in firing neuron B, some growth process or metabolic change takes place in both cells such that A's efficiency in firing B is increased.\n\nThat's **Hebb's Rule**, and the pop version is stickier: **neurons that fire together, wire together.**\n\nTwo neurons active at the same time? The connection between them — the **synapse** — physically strengthens. Next time, A triggers B more easily. Do it enough and the path becomes a highway.\n\nRun Thorndike's cat through this. The cat sees the lever (a pattern of neurons firing) and presses it (another pattern firing). They're active together. Their connection thickens. A hundred trials later, seeing the lever *is* pressing the lever — there's a paved road between them.\n\nThat's a habit, described in hardware.\n\nAnd this is where artificial neural networks come from. In a network, the connection strength between two artificial neurons is a number called a **weight.** Learning means adjusting weights so that useful inputs reliably produce useful outputs. Hebb is the reason \"learning = changing connection strengths\" is the default assumption in the entire field.\n\n**One important correction, because this gets flattened a lot:** Hebb's rule is *not* backpropagation, and the difference is worth thirty seconds.\n\nHebb's rule is **local**. A synapse strengthens based only on what its own two neurons are doing right now. No supervisor, no error signal, nobody keeping score.\n\n**Backpropagation** — the algorithm that actually trains modern networks — is **global and error-driven**. It compares the network's output to a desired output, computes how wrong it was, and pushes that error signal backward through every layer, adjusting each weight by how much it contributed to the mistake. Every weight update depends on information from far away in the network.\n\nThese are genuinely different mechanisms. In fact, the gap between them is one of the liveliest open arguments in neuroscience: brains clearly don't run backprop — there's no known biological machinery for shipping a precise error signal backward across a network — yet backprop works spectacularly well. Whether the brain approximates something like it, or does something else entirely, is unresolved.\n\nSo the honest lineage is: **Hebb gave us the idea that memory lives in connection strengths.** Backprop is a much later, mathematically-driven answer to *how to update* those strengths. Same neighborhood, different building.\n\n🎒\n\nIn the backpack:learning is stored in connection weights. Local, physical, incremental.\n\nFifty-one years, four researchers, and the loop is complete:\n\n| Idea | From | What it becomes |\n|---|---|---|\n| Rewards strengthen the actions that earned them | Thorndike, 1898 | the reward signal |\n| Signals predict future reward | Pavlov, 1900s | value |\n| Behavior can be sculpted step by step | Skinner, 1930s–50s | policies, reward shaping |\n| Learning is stored in connection strengths | Hebb, 1949 | weights |\n\nRead that table top to bottom and you have a specification. Something that acts, predicts, gets shaped by consequences, and stores what it learned in the strength of its connections.\n\nNobody had built it. Nobody could — the ideas were all written in the language of biology, describing wet tissue that nobody could manufacture.\n\nBut 1949 is the year that stops being true. The war had just produced digital computers. A new field called **cybernetics** was arguing that control and feedback worked the same way in animals and machines, and that the distinction was less fundamental than everyone assumed.\n\nFor the first time, people had a spec for learning *and* a machine to run it on.\n\nBefore we leave the psychologists, here's the crack in the foundation — worth holding onto, because it takes another twenty years to fix.\n\nEvery idea above says learning happens when things **co-occur.** Metronome and food together → learn. Lever and fish together → learn. Neurons active together → wire together.\n\nBut co-occurrence can't be the whole story:\n\n**Your alarm has gone off at 7am every day for a year.** By the co-occurrence rule, this morning should have strengthened that association a little further, same as every other morning. It didn't. You learned it in the first week and then stopped — even though the pairing kept happening, perfectly, hundreds more times.\n\nSo what switches learning *off*?\n\nNobody in this post can answer that. The answer arrives in 1972, from a pair of psychologists still studying rats, and when it does it turns out to be the single most important equation in this entire series. We'll get there.\n\n**RL #2 — Cybernetics and Early Machines (1948–1954).** The first people to actually build this thing. Alan Turing sketching a \"pleasure-pain system\" in 1948. Claude Shannon's Theseus, a mechanical mouse that learned a maze by trial and error and could remember it. And Marvin Minsky wiring together the SNARC — 40 vacuum tubes, motors, and a clutch, arguably the first machine that learned by reinforcement.\n\nThe cats gave us the algorithm. Now watch people try to solder it.\n\n*If you spot an error or think I've oversimplified something, drop it in the comments — I'd rather fix it than have it sit there.*", "url": "https://wpnews.pro/news/rl-1-biological-foundations-and-the-law-of-effect-1898-1949", "canonical_source": "https://dev.to/mitanshgor/rl-1-biological-foundations-and-the-law-of-effect-1898-1949-1a9f", "published_at": "2026-09-02 22:58:37+00:00", "updated_at": "2026-09-02 23:23:37.120959+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["Edward Thorndike"], "alternates": {"html": "https://wpnews.pro/news/rl-1-biological-foundations-and-the-law-of-effect-1898-1949", "markdown": "https://wpnews.pro/news/rl-1-biological-foundations-and-the-law-of-effect-1898-1949.md", "text": "https://wpnews.pro/news/rl-1-biological-foundations-and-the-law-of-effect-1898-1949.txt", "jsonld": "https://wpnews.pro/news/rl-1-biological-foundations-and-the-law-of-effect-1898-1949.jsonld"}}