# RL 2: The first physical and computational RL machines(1948–1954) > Source: > Published: 2026-09-12 05:57:48+00:00 Blog 1 established the *rules* of learning from biology. This era asked the harder question: can we build a machine that actually does it? And so, in basements and university labs, researchers started wiring things up. What followed was a decade of beautiful, clunky, brilliant hardware — machines that stumbled through mazes, adjusted their own connections, and learned, in the most primitive sense, from mistakes. None of them knew they were inventing Reinforcement Learning. They were just trying to make something move. Before anyone wrote "Reinforcement Learning" on a whiteboard, there was **Cybernetics**. In 1948, Norbert Wiener published *Cybernetics: Or Control and Communication in the Animal and the Machine* and made a claim that was radical for its time: the same mathematical principle governs both living organisms and machines — **negative feedback**. Every cybernetic system obsesses over one number: the gap between where it is and where it wants to be. Where `T` is the target state and `S` is the current state. If `D \neq 0`, the system fires an actuation signal to close it. Sound familiar? This difference `D` is the direct ancestor of the **Temporal Difference (TD) error** in modern RL — the signal that tells an agent "something went wrong, correct it." Every gradient update in every deep RL system today is, at its core, a Wiener-style error minimization. **Wiener's punchline:** Intelligence is not magic. It's error minimization on a loop. If you can define a goal and measure distance from it, a machine can "learn" to reach it. In the same year, Alan Turing was thinking about the same problem from a completely different angle. His 1948 paper *Intelligent Machinery* introduced the **P-type (Pain-type) Unorganized Machine** — and it reads like the first design document for a neural network. Turing's argument was blunt: a human infant's cortex is largely a random network that gets organized through experience. He proposed building something equivalent out of NAND gates — a network of units, each with two inputs and one output, each carrying a "configuration state" (`c_1` or `c_2`) that determined its behavior. Learning worked through two signals: **Pain** — when the machine produced wrong output, a stochastic state transition fired across the active gates: **Pleasure** — when the machine succeeded, configurations froze. The probability of any state change dropped to zero: Turing understood immediately that pure random search explodes. If a machine has `n` units each with `k` possible configurations, the search space is `k^n` — intractable even for modest machines. His solution: a **teacher** provides the Pleasure signal to guide the machine toward useful sub-modules before Pain prunes the remaining errors. That's not just a neat trick. It's the origin of **Supervised Fine-Tuning (SFT)** and **Imitation Learning** — ideas that wouldn't be formally named for another 50 years. Turing had quietly shifted the burden of intelligence from the *programmer* to the *environment*. His work stayed obscure for decades. But the DNA is unmistakable: a random network, shaped by reward and punishment signals, converging on useful behavior. Every RLHF pipeline you use today is running the same loop Turing sketched on paper in 1948. While Turing's work stayed on paper, Marvin Minsky and his Princeton colleague Dean Edmonds went to the workshop and built something. In 1951, they constructed the **SNARC — Stochastic Neural Analog Reinforcement Calculator** — the first machine to physically implement Hebbian learning and reinforcement in hardware. Minsky later described it in his 1954 thesis. The SNARC simulated a rat navigating a maze using **40 artificial synapses**. Each synapse was a vacuum tube paired with a motor-driven **potentiometer** — a variable resistor whose resistance could be physically turned up or down. That potentiometer *was* the synaptic weight. Here's how learning happened: This is **credit assignment in hardware**. The machine had no explicit rule saying "this connection caused the reward." It used the propagation of the reward signal backward through the active path to decide which potentiometers to turn. Which is, structurally, what backpropagation does — just with calculus instead of motors. The SNARC couldn't scale. 40 synapses is almost nothing, and the analog components drifted over time. But it demonstrated something profound: **a machine could modify its own internal structure in response to reward.** Not by following a program, but by physically changing the weights of its connections. That idea — adjustable weights updated by reward — is the founding premise of every neural network trained by RL today. Claude Shannon's contribution to this era was more immediately tangible. In 1952, he built **Theseus** — a copper mouse on a 5×5 grid, backed by a cabinet of 75 telephone relays. The mouse had no brain. The relays *were* the brain. Each cell in the 25-cell maze corresponded to a relay configuration that stored which directions had already failed. When the mouse hit a wall, the relay for that direction at that position flipped — eliminating that option and pivoting to the next. When a successful path was found, its relay configurations locked in. The result: place Theseus anywhere in the maze it had already explored, and it went straight to the goal. Place it somewhere new, and it explored from scratch — then remembered. Shannon had built a physical **Look-Up Table** — a direct mapping from states (maze positions) to actions (directions), stored in the mechanical state of 75 relays. This is the hardware ancestor of a Q-table. But the more interesting feature was the **"Anti-Neurotic" circuit**. Shannon noticed that a maze-solving strategy based purely on elimination could trap the mouse in loops — try North, fail; try East, fail; loop back to North infinitely. The anti-neurotic circuit detected circular behavior and injected a random direction change to break the loop. This is the first deliberate hardware implementation of **exploration vs. exploitation** — the tension at the center of almost every RL algorithm written since. The mouse needed to *exploit* what it knew (the locked-in successful directions) but *explore* when stuck. Shannon solved it with a random kick. Modern RL solves it with `\epsilon`-greedy, UCB, or entropy bonuses. Same problem, different tools. Theseus proved that "learning" could be reduced to hardware state changes. No probability distributions, no gradients — just relays flipping. Intelligence as a look-up table. By the mid-50s, the hardware era was showing its limits. Shannon's mouse solved one maze. Minsky's SNARC had 40 synapses. Turing's machines existed mostly on paper. The machines were impressive — but each one was a custom solution to a custom problem. Researchers like **Richard Bellman** grew impatient with this. Wiring up a maze-solver was a parlor trick. What they wanted was a *general* rule: something that could solve any problem involving a long chain of decisions, regardless of what the environment looked like. The ambition shifted from *"can we make this work?"* to *"can we prove the best possible way to act?"* Bellman's answer was the **Principle of Optimality** and the **Bellman Equation** — the idea that the value of being in a state is simply the immediate reward plus the discounted value of wherever you end up next. He translated the mechanical "vibes" of the SNARC and Theseus into pure mathematics: a recursive formula that compressed the entire future of an agent's behavior into a single equation. That equation — and the computational nightmare it revealed, which Bellman called the **Curse of Dimensionality** — is what Blog 3 is about. For now, the key point is this: after a decade of building things, the field realized it needed math to go further. The wires were beautiful. But equations scale. What this era actually established — not as a summary, but as a thesis: **Wiener (1948)** gave learning a *drive*: minimize error on a feedback loop. **Turing (1948)** gave learning a *mechanism*: reward stabilizes, punishment randomizes. **Minsky (1951/54)** gave learning *physical weight*: the first adjustable synapses in hardware. **Shannon (1952)** gave learning *memory*: state-action mappings stored in relay configurations. **Bellman (1957)** gave learning *math*: a recursive formula that would unify all of it — covered in Blog 3. In less than a decade, "learning" went from a biological observation to a wired, probabilistic, physically instantiated process. The machines were small and fragile. But the ideas were load-bearing.