{"slug": "making-reinforcement-learning-lectures-interactive", "title": "Making reinforcement learning lectures interactive", "summary": "A developer writing at yad.codes has published an interactive reinforcement learning course comprising 34 lectures across eight tracks, nine coding exercises with tests, and a checkpoint at the end of every lecture. The course runs entirely on a laptop CPU, spanning value iteration on a gridworld through tabular Q-learning on FrozenLake, PPO with GAE on CartPole, DPO on a toy preference dataset, and GRPO on a verifiable arithmetic task, with exercises backed by graduated hints and reference solutions. The material extends to how language models are trained with RL, including distributed rollouts, inference engines and hardware.", "body_md": "The red dot starts out knowing nothing about this grid. Press Run and it wanders, falls into the pits, and now and then finds the +1. Every step updates its estimate of how much each move is worth. By fifty episodes it heads straight for the goal.\n\nThat's Q-learning. Lecture 03 explains it, and in exercise 03 you write it yourself. From there the guide goes all the way to how language models are trained with RL: 34 lectures in eight tracks, nine exercises with tests, and a checkpoint at the end of every lecture.\n\nepisodes 0reached the goal in the last 20: 0last return 0.00\n\nFigure 1. Q-learning on a small grid, running in your browser. The shading is how much the agent expects to collect from each square, green for good and red for bad, and the arrows are its best move so far. The line underneath is the return of each episode. Run shows every step; Train 50 episodes skips ahead.\n\nmore projects like this\n\nI write up what I build and learn on yad.codes. Sign up to get the next one by email.\n\nSubscribe\n\nOne email to confirm. Unsubscribe anytime.\n\n1One topic, start to finish\n\nEvery lecture is studied the same way: read it, watch it, build it, then check yourself with the page closed. Here are the four steps for the Q-learning in Figure 1.\n\n1\n\nRead\n\nLecture 03 starts from value functions, builds the update one term at a time, shows that it's the Bellman equation rewritten as an update, and then scales it up to DQN.\n\nQ(s, a) ← Q(s, a) + α [ r + γ maxa′ Q(s′, a′) − Q(s, a) ]\n\nOpen exercise 03. You fill in three TODOs in starter.py, and this is one of them:\n\n``` python\ndef q_update(q_table, state, action, reward, next_state, done, alpha: float, gamma: float = GAMMA) -> float:\n    \"\"\"One Q-learning update, in place on `q_table`.\n\n        target = reward                                         if done\n                 reward + gamma * max_a' q_table[next_state, a']  otherwise\n        q_table[state, action] += alpha * (target - q_table[state, action])\n\n    Return the TD error: target - (old q_table[state, action]).\n    \"\"\"\n    # TODO\n    raise NotImplementedError(\"q_update\")\n```\n\nThen run pytest exercises/03-q-learning/. The five tests pass when the code is right. Stuck? The hints go from a nudge to nearly the answer, and you take them one at a time.\n\n4\n\nCheck\n\nClose the page and answer the lecture's checkpoint:\n\nExplain when you'd reach for a value-based method instead of a policy-gradient one.\n\nThen answer it again a week later, without looking.\n\n2Nine things you'll build\n\nEach exercise is a starter file with TODOs, tests that pass only when the code is right, graduated hints, and a reference solution. All nine run on a laptop CPU, from value iteration on a gridworld to GRPO on a toy task with verifiable answers.\n\n01-mdpsValue iteration on a gridworld.5 tests · with lecture 01\n\n03-q-learningTabular Q-learning on FrozenLake.5 tests · with lecture 03\n\n04-actor-criticActor-critic with a learned baseline, on CartPole.14 tests · with lecture 04\n\n05-ppoPPO with GAE, solve CartPole.13 tests · with lecture 06\n\n09-reward-modelBradley-Terry reward model on synthetic preferences.5 tests · with lecture 09\n\n11-dpoDPO on a toy preference dataset.6 tests · with lecture 11\n\n15-grpo-rlvrGRPO on a verifiable arithmetic toy task.16 tests · with lecture 15\n\n20-explorationRND on a sparse-reward chain MDP.13 tests · with lecture 20\n\n3Eight tracks\n\nEvery track builds on Foundations. After that the tracks split by where the reward comes from, and you can take them in the order you need. Each track has its own page, with its lectures in order and the checkpoint for each.\n\nstart here, then pick a track ↓\n\nFigure 2. The eight tracks. A solid arrow means a track's first lecture needs the track it comes from; a dashed one marks a single lecture that builds on another track. Click a track to open it.\n\nHow LLM-RL training runs: distributed rollouts and learners, inference engines, hardware.\n\n3 lectures, about 7 hours\n\n4One MDP, six families\n\nEverything in the lecture series is the same underlying object: an MDP, where an agent picks actions and some signal tells it whether things are going well. What changes between sub-fields is mostly what that signal is and who provides it. Classical RL gets a reward from the environment. RLHF infers a reward from human preference labels. RLAIF replaces the human with an LLM judge or a written constitution. RLVR skips the learned reward model entirely and uses a verifier: a checker for math, a test suite for code. Agentic RL puts the model in a multi-turn loop with an environment that tells it whether the task ultimately succeeded. Offline RL works from logged data only, no fresh interaction.\n\nThe map below shows where each family fits. The lectures fill in the details; CURRICULUM.md is the suggested order.\n\nEvery family below is the same MDP. They differ in three answers: where the reward comes from, where the training data comes from, and how many steps pass before the reward arrives.\n\nReading about RL and getting a policy to learn are different skills. The lectures cover the first. The exercises and checkpoints cover the second, if you use them like this.\n\nWrite the code yourself.Every exercise has a starter.py with TODOs and tests that only pass when the code is right. Fill it in before you look at anything else.\n\nTake one hint at a time.HINTS.md goes from a nudge to nearly the answer. Read the next hint only when you are stuck on the current one.\n\nDo the checkpoint with the book closed.Every lecture ends with something to derive, write, or explain. Close the page first.\n\nPassing is not understanding.An exercise is done when the tests pass and you can say why your implementation works.\n\nDebug small.Use CartPole or a gridworld, where a run takes minutes. Broken RL code often runs without errors and just does not learn.\n\nCome back a week later.Redo an old checkpoint after a gap. Testing yourself after a delay does more for memory than rereading (Roediger and Karpicke, 2006).\n\nPair text with video.Eleven lectures link the matching episode of David Silver's course or Berkeley's CS 285.\n\nRead drafts like drafts.Every lecture here is unreviewed. When a formula matters, check it against the cited paper first.\n\n6Everything here\n\nThe Library lists every page on one page. Search with Ctrl K, or open the graph to see how the pages link to each other.\n\nI'm Yad Konrad. I started this repo on 31 December 2016 as a list of the talks and courses I was learning reinforcement learning from. It grew into the notes, lectures, and exercises on this site.\n\n2016\n\nThe first list. Joelle Pineau's and Pieter Abbeel's talks, Sutton and Barto's book, David Silver's course, and Berkeley's CS 294. It is still on the Watch page.\n\n2017\n\nA car that drives itself in GTA V. That was my way into RL. It became Nvidia-Autopilot-Keras, a Keras version of NVIDIA's end-to-end self-driving model, trained on a GTA V driving dataset.\n\n2017\n\nBerkeley's CS 294. Deep Reinforcement Learning, taught by Sergey Levine, John Schulman, and Chelsea Finn. My notes from it are kept as I wrote them.\n\nlater\n\nProgram synthesis, then NLP with RL, then language models.\n\n2025\n\nThe lectures. 34 lectures, exercises with tests, and cheat sheets, drafted with AI coding assistants and then edited. Each lecture stays marked unreviewed until someone has read it end to end and checked it.\n\n2026\n\nThis site. The lectures, the exercises, the 2017 notes, and the first list, in one place.", "url": "https://wpnews.pro/news/making-reinforcement-learning-lectures-interactive", "canonical_source": "https://rl.yad.codes/", "published_at": "2026-11-14 20:51:47+00:00", "updated_at": "2026-11-14 21:23:18.055831+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-research", "large-language-models"], "entities": ["yad.codes", "FrozenLake", "CartPole", "DQN", "PPO", "GRPO", "DPO", "Bradley-Terry"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/making-reinforcement-learning-lectures-interactive", "markdown": "https://wpnews.pro/news/making-reinforcement-learning-lectures-interactive.md", "text": "https://wpnews.pro/news/making-reinforcement-learning-lectures-interactive.txt", "jsonld": "https://wpnews.pro/news/making-reinforcement-learning-lectures-interactive.jsonld"}}