GRPO from Scratch: Group Relative Policy Optimization in Python

Group Relative Policy Optimization (GRPO) eliminates the value model used in Proximal Policy Optimization (PPO), reducing active memory footprint from 42 GB to 28 GB for a 7B parameter model in FP16 by generating G responses per prompt and normalizing rewards within the group. DeepSeek-R1 was trained with GRPO, enabling reasoning model training without separate reward model rollout infrastructure, according to a technical article by an anonymous author.

Member-only story GRPO from Scratch: Group Relative Policy Optimization in Python Proximal Policy Optimization PPO requires four active components policy, reference, reward, value ; the 42 GB calculation below counts only the three trainable / host-resident weight copies policy, reference, value , since the reward model is typically a frozen scorer that can be off-loaded for a 7B in FP16 16-bit floating point , and forces synchronous rollouts between policy and critic. Group Relative Policy Optimization GRPO eliminates the value model — bringing the active footprint down to 28 GB — by generating G responses to the same prompt, normalizing rewards within the group to compute advantages, and applying the PPO clipping objective directly. No critic. No value function. The group is the baseline. DeepSeek-R1 DeepSeek-AI 2024, DeepSeek-R1: Reasoning Model Trained via Reinforcement Learning was trained with GRPO. It is how you get a reasoning model without a separate reward model rollout infrastructure. Disclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information. Why GRPO Over PPO PPO Proximal Policy Optimization for language models requires four active components during training: