{"slug": "rl-2-the-first-physical-and-computational-rl-machines-1948-1954", "title": "RL 2: The first physical and computational RL machines(1948–1954)", "summary": "A historical survey traces the first physical and computational reinforcement learning machines built between 1948 and 1954, beginning with Norbert Wiener's 1948 Cybernetics and its negative-feedback principle, which the piece calls the direct ancestor of the temporal difference error used in modern RL. It also covers Alan Turing's 1948 P-type Unorganized Machine, a NAND-gate network shaped by pain and pleasure signals that prefigured supervised fine-tuning and imitation learning, and the 1951 SNARC built by Marvin Minsky and Dean Edmonds, which used 40 artificial synapses of vacuum tubes and motor-driven potentiometers to simulate a rat navigating a maze.", "body_md": "Blog 1 established the *rules* of learning from biology. This era asked the harder question: can we build a machine that actually does it?\n\nAnd so, in basements and university labs, researchers started wiring things up. What followed was a decade of beautiful, clunky, brilliant hardware — machines that stumbled through mazes, adjusted their own connections, and learned, in the most primitive sense, from mistakes. None of them knew they were inventing Reinforcement Learning. They were just trying to make something move.\n\nBefore anyone wrote \"Reinforcement Learning\" on a whiteboard, there was **Cybernetics**. In 1948, Norbert Wiener published *Cybernetics: Or Control and Communication in the Animal and the Machine* and made a claim that was radical for its time: the same mathematical principle governs both living organisms and machines — **negative feedback**.\n\nEvery cybernetic system obsesses over one number: the gap between where it is and where it wants to be.\n\nWhere `T` is the target state and `S` is the current state. If `D \\neq 0`, the system fires an actuation signal to close it. Sound familiar? This difference `D` is the direct ancestor of the **Temporal Difference (TD) error** in modern RL — the signal that tells an agent \"something went wrong, correct it.\" Every gradient update in every deep RL system today is, at its core, a Wiener-style error minimization.\n\n**Wiener's punchline:** Intelligence is not magic. It's error minimization on a loop. If you can define a goal and measure distance from it, a machine can \"learn\" to reach it.\n\nIn the same year, Alan Turing was thinking about the same problem from a completely different angle. His 1948 paper *Intelligent Machinery* introduced the **P-type (Pain-type) Unorganized Machine** — and it reads like the first design document for a neural network.\n\nTuring's argument was blunt: a human infant's cortex is largely a random network that gets organized through experience. He proposed building something equivalent out of NAND gates — a network of units, each with two inputs and one output, each carrying a \"configuration state\" (`c_1` or `c_2`) that determined its behavior.\n\nLearning worked through two signals:\n\n**Pain** — when the machine produced wrong output, a stochastic state transition fired across the active gates:\n\n**Pleasure** — when the machine succeeded, configurations froze. The probability of any state change dropped to zero:\n\nTuring understood immediately that pure random search explodes. If a machine has `n` units each with `k` possible configurations, the search space is `k^n` — intractable even for modest machines. His solution: a **teacher** provides the Pleasure signal to guide the machine toward useful sub-modules before Pain prunes the remaining errors.\n\nThat's not just a neat trick. It's the origin of **Supervised Fine-Tuning (SFT)** and **Imitation Learning** — ideas that wouldn't be formally named for another 50 years. Turing had quietly shifted the burden of intelligence from the *programmer* to the *environment*.\n\nHis work stayed obscure for decades. But the DNA is unmistakable: a random network, shaped by reward and punishment signals, converging on useful behavior. Every RLHF pipeline you use today is running the same loop Turing sketched on paper in 1948.\n\nWhile Turing's work stayed on paper, Marvin Minsky and his Princeton colleague Dean Edmonds went to the workshop and built something.\n\nIn 1951, they constructed the **SNARC — Stochastic Neural Analog Reinforcement Calculator** — the first machine to physically implement Hebbian learning and reinforcement in hardware. Minsky later described it in his 1954 thesis.\n\nThe SNARC simulated a rat navigating a maze using **40 artificial synapses**. Each synapse was a vacuum tube paired with a motor-driven **potentiometer** — a variable resistor whose resistance could be physically turned up or down. That potentiometer *was* the synaptic weight.\n\nHere's how learning happened:\n\nThis is **credit assignment in hardware**. The machine had no explicit rule saying \"this connection caused the reward.\" It used the propagation of the reward signal backward through the active path to decide which potentiometers to turn. Which is, structurally, what backpropagation does — just with calculus instead of motors.\n\nThe SNARC couldn't scale. 40 synapses is almost nothing, and the analog components drifted over time. But it demonstrated something profound: **a machine could modify its own internal structure in response to reward.** Not by following a program, but by physically changing the weights of its connections.\n\nThat idea — adjustable weights updated by reward — is the founding premise of every neural network trained by RL today.\n\nClaude Shannon's contribution to this era was more immediately tangible. In 1952, he built **Theseus** — a copper mouse on a 5×5 grid, backed by a cabinet of 75 telephone relays.\n\nThe mouse had no brain. The relays *were* the brain. Each cell in the 25-cell maze corresponded to a relay configuration that stored which directions had already failed. When the mouse hit a wall, the relay for that direction at that position flipped — eliminating that option and pivoting to the next. When a successful path was found, its relay configurations locked in.\n\nThe result: place Theseus anywhere in the maze it had already explored, and it went straight to the goal. Place it somewhere new, and it explored from scratch — then remembered.\n\nShannon had built a physical **Look-Up Table** — a direct mapping from states (maze positions) to actions (directions), stored in the mechanical state of 75 relays. This is the hardware ancestor of a Q-table.\n\nBut the more interesting feature was the **\"Anti-Neurotic\" circuit**. Shannon noticed that a maze-solving strategy based purely on elimination could trap the mouse in loops — try North, fail; try East, fail; loop back to North infinitely. The anti-neurotic circuit detected circular behavior and injected a random direction change to break the loop.\n\nThis is the first deliberate hardware implementation of **exploration vs. exploitation** — the tension at the center of almost every RL algorithm written since. The mouse needed to *exploit* what it knew (the locked-in successful directions) but *explore* when stuck. Shannon solved it with a random kick. Modern RL solves it with `\\epsilon`-greedy, UCB, or entropy bonuses. Same problem, different tools.\n\nTheseus proved that \"learning\" could be reduced to hardware state changes. No probability distributions, no gradients — just relays flipping. Intelligence as a look-up table.\n\nBy the mid-50s, the hardware era was showing its limits. Shannon's mouse solved one maze. Minsky's SNARC had 40 synapses. Turing's machines existed mostly on paper. The machines were impressive — but each one was a custom solution to a custom problem.\n\nResearchers like **Richard Bellman** grew impatient with this. Wiring up a maze-solver was a parlor trick. What they wanted was a *general* rule: something that could solve any problem involving a long chain of decisions, regardless of what the environment looked like.\n\nThe ambition shifted from *\"can we make this work?\"* to *\"can we prove the best possible way to act?\"*\n\nBellman's answer was the **Principle of Optimality** and the **Bellman Equation** — the idea that the value of being in a state is simply the immediate reward plus the discounted value of wherever you end up next. He translated the mechanical \"vibes\" of the SNARC and Theseus into pure mathematics: a recursive formula that compressed the entire future of an agent's behavior into a single equation.\n\nThat equation — and the computational nightmare it revealed, which Bellman called the **Curse of Dimensionality** — is what Blog 3 is about. For now, the key point is this: after a decade of building things, the field realized it needed math to go further.\n\nThe wires were beautiful. But equations scale.\n\nWhat this era actually established — not as a summary, but as a thesis:\n\n**Wiener (1948)** gave learning a *drive*: minimize error on a feedback loop.\n\n**Turing (1948)** gave learning a *mechanism*: reward stabilizes, punishment randomizes.\n\n**Minsky (1951/54)** gave learning *physical weight*: the first adjustable synapses in hardware.\n\n**Shannon (1952)** gave learning *memory*: state-action mappings stored in relay configurations.\n\n**Bellman (1957)** gave learning *math*: a recursive formula that would unify all of it — covered in Blog 3.\n\nIn less than a decade, \"learning\" went from a biological observation to a wired, probabilistic, physically instantiated process. The machines were small and fragile. But the ideas were load-bearing.", "url": "https://wpnews.pro/news/rl-2-the-first-physical-and-computational-rl-machines-1948-1954", "canonical_source": "https://dev.to/mitanshgor/rl-2-the-first-physical-and-computational-rl-machines1948-1954-1amh", "published_at": "2026-09-12 05:57:48+00:00", "updated_at": "2026-09-12 06:26:28.990681+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "neural-networks"], "entities": ["Norbert Wiener", "Alan Turing", "Marvin Minsky", "Dean Edmonds", "SNARC", "Cybernetics", "Princeton"], "alternates": {"html": "https://wpnews.pro/news/rl-2-the-first-physical-and-computational-rl-machines-1948-1954", "markdown": "https://wpnews.pro/news/rl-2-the-first-physical-and-computational-rl-machines-1948-1954.md", "text": "https://wpnews.pro/news/rl-2-the-first-physical-and-computational-rl-machines-1948-1954.txt", "jsonld": "https://wpnews.pro/news/rl-2-the-first-physical-and-computational-rl-machines-1948-1954.jsonld"}}