{"slug": "grpo-from-scratch-group-relative-policy-optimization-in-python", "title": "GRPO from Scratch: Group Relative Policy Optimization in Python", "summary": "Group Relative Policy Optimization (GRPO) eliminates the value model used in Proximal Policy Optimization (PPO), reducing active memory footprint from 42 GB to 28 GB for a 7B parameter model in FP16 by generating G responses per prompt and normalizing rewards within the group. DeepSeek-R1 was trained with GRPO, enabling reasoning model training without separate reward model rollout infrastructure, according to a technical article by an anonymous author.", "body_md": "Member-only story\n\n# GRPO from Scratch: Group Relative Policy Optimization in Python\n\nProximal Policy Optimization (PPO) requires four active components (policy, reference, reward, value); the 42 GB calculation below counts only the three trainable / host-resident weight copies (policy, reference, value), since the reward model is typically a frozen scorer that can be off-loaded for a 7B in FP16 (16-bit floating point), and forces synchronous rollouts between policy and critic. Group Relative Policy Optimization (GRPO) eliminates the value model — bringing the active footprint down to 28 GB — by generating G responses to the same prompt, normalizing rewards within the group to compute advantages, and applying the PPO clipping objective directly. No critic. No value function. The group is the baseline.\n\nDeepSeek-R1 (DeepSeek-AI 2024, *DeepSeek-R1: Reasoning Model Trained via Reinforcement Learning*) was trained with GRPO. It is how you get a reasoning model without a separate reward model rollout infrastructure.\n\nDisclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information.\n\n## Why GRPO Over PPO\n\nPPO (Proximal Policy Optimization) for language models requires four active components during training:", "url": "https://wpnews.pro/news/grpo-from-scratch-group-relative-policy-optimization-in-python", "canonical_source": "https://pub.towardsai.net/grpo-from-scratch-group-relative-policy-optimization-in-python-c2c5fb1df4b3?source=rss----98111c9905da---4", "published_at": "2026-07-20 13:01:03+00:00", "updated_at": "2026-07-20 13:21:25.755104+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["DeepSeek-R1", "DeepSeek-AI", "Google"], "alternates": {"html": "https://wpnews.pro/news/grpo-from-scratch-group-relative-policy-optimization-in-python", "markdown": "https://wpnews.pro/news/grpo-from-scratch-group-relative-policy-optimization-in-python.md", "text": "https://wpnews.pro/news/grpo-from-scratch-group-relative-policy-optimization-in-python.txt", "jsonld": "https://wpnews.pro/news/grpo-from-scratch-group-relative-policy-optimization-in-python.jsonld"}}