Reinforcement Learning Series

A developer created a series to demystify reinforcement learning by tracing its evolution chronologically, from early psychology and mechanical machines to modern mathematical breakthroughs. The series covers foundational concepts such as the Law of Effect, Pavlovian reinforcement, Hebbian learning, and key milestones like Turing's pleasure-pain system, Bellman's dynamic programming, Q-Learning, and Deep RL with DQN.

I created this series to make that steep learning curve far less daunting for you. My goal is to trace the evolution of RL chronologically—from its roots in early psychology and physical mechanical machines to digital binary systems and modern mathematical breakthroughs. By breaking down complex concepts with clear visual guides, graphics, and real-world analogies, I hope to demystify RL and give back to the community that inspired me. Let’s dive in This entry explores the Law of Effect 1911 , where Edward Thorndike established that actions followed by satisfaction are strengthened. It also covers Ivan Pavlov’s formal definition of reinforcement 1927 and Donald Hebb’s 1949 hypothesis that "neurons that fire together, wire together," laying the groundwork for neural learning. This blog details the first computational investigations, including Alan Turing’s "pleasure-pain system" 1948 and Marvin Minsky’s construction of SNARCs 1954 , the first analog neural-network reinforcement calculators. It also highlights Claude Shannon’s 1952 demonstration of "Theseus," a maze-running mouse that used trial and error to "remember" paths. This post focuses on Richard Bellman’s development of Dynamic Programming and the Bellman Equation 1957 , which provided the formal tools for solving optimal control problems. It also introduces Markov Decision Processes MDPs and Ron Howard’s policy iteration method 1960 , which are the bedrock of modern RL theory. This entry discusses Arthur Samuel’s celebrated checkers program 1959 , which was the first to implement temporal-difference TD ideas by using board evaluation differences to modify earlier positions. It also details Donald Michie’s MENACE 1961 , a physical "matchbox" engine that learned to play Noughts and Crosses through reinforcement. This blog examines the Soviet lineage of RL, specifically Mikhail Tsetlin’s work on "learning automata" 1961 , which were finite-state machines that updated action probabilities based on stochastic environmental feedback. It traces how this research was later systematised by Narendra and Thathachar in 1974. This post covers Harry Klopf’s pivotal role in reviving the trial-and-error thread by proposing the "hedonistic neuron" 1972 , which argued that neurons act as pleasure-seeking units. It also explores Paul Werbos’s 1974 thesis, which was the first to describe backpropagation in the context of "Adaptive Dynamic Programming". This blog highlights the formalisation of modern RL, beginning with Sutton and Barto’s 1981 model of classical conditioning. It details the invention of the Actor-Critic architecture 1983 and Sutton’s 1984 dissertation, which provided the first rigorous computational analysis of temporal credit assignment . This entry documents two watershed moments: Sutton’s 1988 formalisation of TD learning and Chris Watkins’s 1989 introduction of Q-Learning , the first model-free, off-policy algorithm. It also covers the introduction of SARSA 1994 as an on-policy alternative that prioritises training stability. This post details Gerald Tesauro’s TD-Gammon 1992 , which achieved grandmaster-level backgammon play using neural networks and self-play. It also covers the options framework 1999 by Sutton, Precup, and Singh, which introduced temporal abstraction to help RL agents solve problems across different timescales. This entry explains the Policy Gradient Theorem 2000 , which proved that policy parameters could be optimised using only local information. It also introduces Sham Kakade’s Natural Policy Gradient 2002 , a precursor to modern trust-region methods that account for the geometry of parameter space. This post chronicles the birth of Deep RL with DeepMind’s DQN 2013/2015 , which used Experience Replay and Target Networks to master Atari games directly from pixels. It also details DDPG 2015 , which extended these successes to continuous action spaces like robotics. This entry explores Trust Region Policy Optimization TRPO 2015 and its simpler successor, Proximal Policy Optimization PPO 2017 , which became the industry standard for stable training. It also covers the historic victory of AlphaGo over Lee Sedol in 2016. This post details the Rainbow agent 2017 , which combined seven independent DQN improvements into one "super-agent". It also highlights MuZero 2019 , which learned to master games like Go and Chess without even being told the rules of the environment. This blog focuses on Agent57 2020 , the first agent to surpass the human benchmark on all 57 Atari games by solving "hard exploration" problems. It also covers Gato 2022 , a single transformer-based generalist agent capable of playing Atari, captioning images, and controlling robot arms. This entry describes the Decision Transformer 2021 , which reframed RL as a sequence modeling problem, bypassing traditional bootstrapping. It also covers the rise of Reinforcement Learning from Human Feedback RLHF with InstructGPT and the subsequent development of Direct Preference Optimization DPO . This blog explores DeepSeek Math and GRPO 2024 , which removed the need for a separate critic model to improve mathematical reasoning. It details the breakthrough of DeepSeek-R1 2024/2025 , which proved that pure RL can spontaneously elicit self-correction and reflection in language models. This final entry discusses the 2025 NeurIPS Best Paper on 1,000-layer RL networks , which broke the "shallow network" assumption to unlock 50x performance gains. It reflects on Kimi k1.5 and the role of the 2024 Turing Award in cementing RL as a foundational pillar of the AI revolution.