The red dot starts out knowing nothing about this grid. Press Run and it wanders, falls into the pits, and now and then finds the +1. Every step updates its estimate of how much each move is worth. By fifty episodes it heads straight for the goal.
That's Q-learning. Lecture 03 explains it, and in exercise 03 you write it yourself. From there the guide goes all the way to how language models are trained with RL: 34 lectures in eight tracks, nine exercises with tests, and a checkpoint at the end of every lecture.
episodes 0reached the goal in the last 20: 0last return 0.00
Figure 1. Q-learning on a small grid, running in your browser. The shading is how much the agent expects to collect from each square, green for good and red for bad, and the arrows are its best move so far. The line underneath is the return of each episode. Run shows every step; Train 50 episodes skips ahead.
more projects like this
I write up what I build and learn on yad.codes. Sign up to get the next one by email.
One email to confirm. Unsubscribe anytime.
1One topic, start to finish
Every lecture is studied the same way: read it, watch it, build it, then check yourself with the page closed. Here are the four steps for the Q-learning in Figure 1.
1
Read
Lecture 03 starts from value functions, builds the update one term at a time, shows that it's the Bellman equation rewritten as an update, and then scales it up to DQN.
Q(s, a) ← Q(s, a) + α [ r + γ maxa′ Q(s′, a′) − Q(s, a) ]
Open exercise 03. You fill in three TODOs in starter.py, and this is one of them:
def q_update(q_table, state, action, reward, next_state, done, alpha: float, gamma: float = GAMMA) -> float:
"""One Q-learning update, in place on `q_table`.
target = reward if done
reward + gamma * max_a' q_table[next_state, a'] otherwise
q_table[state, action] += alpha * (target - q_table[state, action])
Return the TD error: target - (old q_table[state, action]).
"""
raise NotImplementedError("q_update")
Then run pytest exercises/03-q-learning/. The five tests pass when the code is right. Stuck? The hints go from a nudge to nearly the answer, and you take them one at a time.
4
Check
Close the page and answer the lecture's checkpoint:
Explain when you'd reach for a value-based method instead of a policy-gradient one.
Then answer it again a week later, without looking.
2Nine things you'll build
Each exercise is a starter file with TODOs, tests that pass only when the code is right, graduated hints, and a reference solution. All nine run on a laptop CPU, from value iteration on a gridworld to GRPO on a toy task with verifiable answers.
01-mdpsValue iteration on a gridworld.5 tests · with lecture 01
03-q-learningTabular Q-learning on FrozenLake.5 tests · with lecture 03
04-actor-criticActor-critic with a learned baseline, on CartPole.14 tests · with lecture 04
05-ppoPPO with GAE, solve CartPole.13 tests · with lecture 06
09-reward-modelBradley-Terry reward model on synthetic preferences.5 tests · with lecture 09
11-dpoDPO on a toy preference dataset.6 tests · with lecture 11
15-grpo-rlvrGRPO on a verifiable arithmetic toy task.16 tests · with lecture 15
20-explorationRND on a sparse-reward chain MDP.13 tests · with lecture 20
3Eight tracks
Every track builds on Foundations. After that the tracks split by where the reward comes from, and you can take them in the order you need. Each track has its own page, with its lectures in order and the checkpoint for each.
start here, then pick a track ↓
Figure 2. The eight tracks. A solid arrow means a track's first lecture needs the track it comes from; a dashed one marks a single lecture that builds on another track. Click a track to open it.
How LLM-RL training runs: distributed rollouts and learners, inference engines, hardware.
3 lectures, about 7 hours
4One MDP, six families
Everything in the lecture series is the same underlying object: an MDP, where an agent picks actions and some signal tells it whether things are going well. What changes between sub-fields is mostly what that signal is and who provides it. Classical RL gets a reward from the environment. RLHF infers a reward from human preference labels. RLAIF replaces the human with an LLM judge or a written constitution. RLVR skips the learned reward model entirely and uses a verifier: a checker for math, a test suite for code. Agentic RL puts the model in a multi-turn loop with an environment that tells it whether the task ultimately succeeded. Offline RL works from logged data only, no fresh interaction.
The map below shows where each family fits. The lectures fill in the details; CURRICULUM.md is the suggested order.
Every family below is the same MDP. They differ in three answers: where the reward comes from, where the training data comes from, and how many steps pass before the reward arrives.
Reading about RL and getting a policy to learn are different skills. The lectures cover the first. The exercises and checkpoints cover the second, if you use them like this.
Write the code yourself.Every exercise has a starter.py with TODOs and tests that only pass when the code is right. Fill it in before you look at anything else.
Take one hint at a time.HINTS.md goes from a nudge to nearly the answer. Read the next hint only when you are stuck on the current one.
Do the checkpoint with the book closed.Every lecture ends with something to derive, write, or explain. Close the page first.
Passing is not understanding.An exercise is done when the tests pass and you can say why your implementation works.
Debug small.Use CartPole or a gridworld, where a run takes minutes. Broken RL code often runs without errors and just does not learn.
Come back a week later.Redo an old checkpoint after a gap. Testing yourself after a delay does more for memory than rereading (Roediger and Karpicke, 2006).
Pair text with video.Eleven lectures link the matching episode of David Silver's course or Berkeley's CS 285.
Read drafts like drafts.Every lecture here is unreviewed. When a formula matters, check it against the cited paper first.
6Everything here
The Library lists every page on one page. Search with Ctrl K, or open the graph to see how the pages link to each other.
I'm Yad Konrad. I started this repo on 31 December 2016 as a list of the talks and courses I was learning reinforcement learning from. It grew into the notes, lectures, and exercises on this site.
2016
The first list. Joelle Pineau's and Pieter Abbeel's talks, Sutton and Barto's book, David Silver's course, and Berkeley's CS 294. It is still on the Watch page.
2017
A car that drives itself in GTA V. That was my way into RL. It became Nvidia-Autopilot-Keras, a Keras version of NVIDIA's end-to-end self-driving model, trained on a GTA V driving dataset.
2017
Berkeley's CS 294. Deep Reinforcement Learning, taught by Sergey Levine, John Schulman, and Chelsea Finn. My notes from it are kept as I wrote them.
later
Program synthesis, then NLP with RL, then language models.
2025
The lectures. 34 lectures, exercises with tests, and cheat sheets, drafted with AI coding assistants and then edited. Each lecture stays marked unreviewed until someone has read it end to end and checked it.
2026
This site. The lectures, the exercises, the 2017 notes, and the first list, in one place.