Member-only story
Proximal Policy Optimization (PPO) requires four active components (policy, reference, reward, value); the 42 GB calculation below counts only the three trainable / host-resident weight copies (policy, reference, value), since the reward model is typically a frozen scorer that can be off-loaded for a 7B in FP16 (16-bit floating point), and forces synchronous rollouts between policy and critic. Group Relative Policy Optimization (GRPO) eliminates the value model — bringing the active footprint down to 28 GB — by generating G responses to the same prompt, normalizing rewards within the group to compute advantages, and applying the PPO clipping objective directly. No critic. No value function. The group is the baseline.
DeepSeek-R1 (DeepSeek-AI 2024, DeepSeek-R1: Reasoning Model Trained via Reinforcement Learning) was trained with GRPO. It is how you get a reasoning model without a separate reward model rollout infrastructure.
Disclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information.
Why GRPO Over PPO #
PPO (Proximal Policy Optimization) for language models requires four active components during training: