{"slug": "i-implemented-the-algorithm-behind-chatgpt-from-scratch-day-8-ppo", "title": "I Implemented the Algorithm Behind ChatGPT From Scratch - Day 8 (PPO).", "summary": "A developer implementing reinforcement learning algorithms from scratch in JAX reports that PPO (Proximal Policy Optimization), the algorithm behind ChatGPT's RLHF training, is simpler than expected. The developer found that PPO's core innovation is a single clipping operation that limits policy updates, preventing training collapse, and that it adds multiple epochs per batch and an entropy bonus over standard Actor-Critic. The developer's experiments on a gridworld showed the critic successfully mapped the environment's safe and dangerous states, and the learned policy matched results from previous algorithms, highlighting the underlying Bellman intuition.", "body_md": "SERIES: Learning RL and JAX in Public - from zero to DeepMind :)\n\nWhen people ask \"how was ChatGPT trained?\", the answer usually involves RLHF - Reinforcement Learning from Human Feedback. And the RL part of RLHF is PPO.\n\nDay 8 is PPO. And it turns out to be simpler than I expected.\n\n**The problem that PPO solves**\n\nActor-Critic (Day 7) worked, but it had an instability problem. A single episode with an unusually high or low return would push the network weights too far in one direction. The policy would change dramatically. Sometimes it would unlearn things it already knew. Training would collapse.\n\nThe fix is elegant: put a speed limit on learning.\n\n**The one idea that is PPO**\n\nAfter each update, PPO compares the new policy to the old one:\n\n```\nratio = new_probability / old_probability\n```\n\nIf ratio = 1.0: nothing changed for this action.\n\nIf ratio = 1.5: the new policy is 50% more likely to take this action.\n\nIf ratio = 0.5: the new policy is 50% less likely.\n\nThen PPO clips this ratio. It says: I will not let you change by more than 20% in one update.\n\n```\nclipped_ratio = clip(ratio, 0.8, 1.2)\nloss = -min(ratio * advantage, clipped_ratio * advantage)\n```\n\nThat is literally it. One clip. One min. That is the entirety of what makes PPO different from Actor-Critic.\n\nThe gradient from a clipped update becomes zero once the ratio hits the boundary. The policy stops updating further for that action in that step. Come back next batch and nudge it again if you want more.\n\nSmall stable steps, every update.\n\n**The three things PPO adds over Actor-Critic**\n\n**1. Clipping (the main idea)**\n\nKeeps the policy from changing too much in one shot.\n\n**2. Multiple epochs per batch**\n\nActor-Critic uses each episode once. PPO collects a batch of episodes and then runs 4 gradient updates on that same data. More learning per episode. More sample efficient.\n\n**3. Entropy bonus**\n\nEntropy measures how spread out the action probabilities are. High entropy means the agent is still considering many options. Low entropy means it has collapsed to always picking one action.\n\nPPO adds a small reward for entropy:\n\n```\ntotal_loss = policy_loss + critic_loss - 0.01 * entropy\n```\n\nThis keeps the agent exploring longer before committing. Without it, policies can collapse to one action too early and get stuck.\n\n**What the training loop looks like**\n\n```\nfor iteration in range(200):\n    # collect 10 episodes with current policy\n    batch = collect_batch(actor, critic, episodes=10)\n\n    # PPO epochs: squeeze 4 updates out of this batch\n    for epoch in range(4):\n        actor_loss = clipped_ppo_loss(batch)\n        critic_loss = value_loss(batch)\n        update(actor, critic)\n```\n\nTwo loops. Outer loop collects data. Inner loop extracts learning from that data. Actor-Critic had one loop. This is the reason PPO trains faster per episode collected.\n\n**What I saw when I ran this**\n\nThe value estimates from the critic after training:\n\n```\nstates near goal:  +0.6 to +0.8\nsafe middle path:  +0.1 to +0.4\nstates near holes: -0.3 to -0.6\n```\n\nThe critic mapped out the whole gridworld just by watching the actor fail and succeed. Nobody told it where the holes were. It figured it out.\n\nThe policy arrows lined up cleanly. Same result as Q-learning (Day 4), DQN (Day 5), REINFORCE (Day 6), Actor-Critic (Day 7). Different algorithm every time. Same learned behavior.\n\nThat pattern keeps hitting me. The Bellman intuition runs through all of it.\n\n**Why PPO became the default**\n\nThree reasons: it is stable (the clip prevents collapse), it is simple (one extra line over Actor-Critic), and it is general (discrete actions, continuous actions, LLM fine-tuning - all the same algorithm).\n\nWhen OpenAI trained ChatGPT, they had humans rank responses. Those rankings became a reward signal. PPO optimized the language model against that signal. The RL loop you just implemented is the same loop, just with a very different environment and a very different reward function.\n\nWhen I built the forge project (a GRPO trainer), I did not fully understand why it worked the way it did. GRPO is PPO with one change: instead of a single critic estimating advantage, it runs a group of episodes and compares their returns to each other. Same clipping. Same stability. The group comparison replaces the critic.\n\nFive days ago I could not have explained that. Now I can.\n\nDay 9: Haiku. DeepMind's neural network library. We rewrite all of this in a fraction of the code, and it starts looking like actual research-grade JAX.\n\nAll code from this series, organised by day, is on my GitHub: [https://github.com/MadhumithaKolkar/jax-rl-lab](https://github.com/MadhumithaKolkar/jax-rl-lab)\n\nHappy learning everyone !\n\n~ Madhumitha Kolkar (index_0)", "url": "https://wpnews.pro/news/i-implemented-the-algorithm-behind-chatgpt-from-scratch-day-8-ppo", "canonical_source": "https://dev.to/madhumithakolkar/i-implemented-the-algorithm-behind-chatgpt-from-scratch-day-8-ppo-o3f", "published_at": "2026-07-31 16:12:33+00:00", "updated_at": "2026-07-31 16:36:46.471195+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "developer-tools"], "entities": ["OpenAI", "ChatGPT", "JAX", "DeepMind"], "alternates": {"html": "https://wpnews.pro/news/i-implemented-the-algorithm-behind-chatgpt-from-scratch-day-8-ppo", "markdown": "https://wpnews.pro/news/i-implemented-the-algorithm-behind-chatgpt-from-scratch-day-8-ppo.md", "text": "https://wpnews.pro/news/i-implemented-the-algorithm-behind-chatgpt-from-scratch-day-8-ppo.txt", "jsonld": "https://wpnews.pro/news/i-implemented-the-algorithm-behind-chatgpt-from-scratch-day-8-ppo.jsonld"}}