{"slug": "test-time-compute-and-grpo-in-practice-from-ppo-to-critic-free-reinforcement", "title": "Test-Time Compute and GRPO in Practice: From PPO to Critic-Free Reinforcement Learning", "summary": "A developer detailed how frontier LLM development is shifting from pre-training scaling to test-time compute scaling, outlining three architectural regimes: sequential chain-of-thought expansion, leaf-level sampling and voting, and prefix-level search with process reward models. The writeup argues that PPO's four-network setup becomes impractical for reasoning models with 10k+ token outputs, and that DeepSeek-R1's fusion of sequential CoT expansion with critic-free reinforcement learning (GRPO) shows pure rule-based RL can induce deep reasoning without hand-engineered step-level PRMs.", "body_md": "For the past several years, the foundational law of frontier LLM development was Chinchilla's **Pre-training Scaling Laws**: stack deeper transformer layers, ingest multi-trillion token corpora, and burn increasingly massive GPU clusters.\n\nHowever, entering 2026, this brute-force approach has encountered formidable physical and thermodynamic bottlenecks:\n\nAs pre-training scaling slows, frontier reasoning engines (such as OpenAI o1/o3 and DeepSeek-R1) have ignited a secondary growth curve: **Test-Time Compute Scaling Laws**.\n\n```\ngraph LR\n    subgraph Traditional Paradigm: One-Shot Pre-training Inference\n        A1[\"Complex Math/Coding Prompt\"] --> A2[\"70B~400B Dense Base LLM\"] --> A3[\"Greedy Decoding (Prone to Hallucinations)\"]\n    end\n    subgraph Reasoning Paradigm: Test-Time Compute Scaling\n        B1[\"Complex Math/Coding Prompt\"] --> B2[\"Compact Base Model\"] --> B3[\"Extended Chain-of-Thought (CoT)\"] --> B4[\"Self-Verification & Backtracking\"] --> B5[\"Deterministic Accurate Solution\"]\n    end\n```\n\nRather than spending millions of dollars during pre-training to memorize answers to every conceivable question, test-time scaling trains models to allocate dynamic computation at inference time—thinking, calculating, and self-correcting before providing a response.\n\nIn modern literature, extending test-time compute falls into three primary architectural regimes:\n\n| Scaling Regime | Core Mechanism | Primary Compute Bottleneck | Representative Work | Bottlenecks & Failure Modes | \n|---|---|---|---|---|\n| **1. Sequential CoT Expansion** | The model outputs multi-thousand token chains of thought ( `<think> ... </think>` ), enabling backtracking and scratchpad verification. | Autoregressive decoding latency | DeepSeek-R1, OpenAI o1 | Prone to \"overthinking\" loops on trivial prompts; latency increases substantially. | \n| **2. Leaf-level Sampling & Voting** | Parallel sampling of $N$ diverse paths, combined with majority voting or verifiers. | Batch concurrency capacity | Best-of-N, Self-Consistency | Search space is unguided; incorrect trajectories waste full GPU decode cycles. | \n| **3. Prefix-level Search with PRMs** | Process Reward Models (PRMs) score intermediate steps within tree search (Beam Search / MCTS). | Step-level verifier evaluation | AlphaGo-style MCTS, Step-PRMs | Step-level PRM annotations are costly; imperfect verifiers invite \"reward hacking.\" | \n\nThe breakthrough of DeepSeek-R1 lies in fusing **sequential chain-of-thought expansion** with **critic-free reinforcement learning**, proving that pure rule-based RL can induce deep reasoning behaviors without manually engineered step-by-step PRMs.\n\nFor years, **PPO (Proximal Policy Optimization)** was the standard algorithm for post-training alignment. However, when applied to reasoning models with 10k+ token outputs, PPO collapses under severe infrastructure and mathematical constraints.\n\nA classic PPO training setup requires hosting and synchronizing **four distinct neural networks**:\n\n```\ngraph TD\n    subgraph Traditional PPO Architecture\n        P1[\"Actor Model (Trainable)\"]\n        P2[\"Critic Model (Trainable - Massive VRAM)\"]\n        P3[\"Reference Model (Frozen)\"]\n        P4[\"Reward Model (Frozen)\"]\n    end\n    subgraph GRPO Architecture\n        G1[\"Actor Model (Trainable)\"]\n        G2[\"Ref Weights / Analytical KL Calculation\"]\n        G3[\"Deterministic Environment (Python Sandbox / Unit Tests / Matcher)\"]\n    end\n```\n\nIn a 70B parameter setup, loading the Actor and Critic alongside their respective AdamW optimizer states easily demands over **600GB of VRAM**. This forces teams to deploy complex tensor and pipeline parallelism merely to fit the training loop. For low-level driver and memory bus topology guidelines, consult our [NVIDIA GPU Package Architecture Deep Dive](https://blog.llmgo.top/en/articles/nvidia-gpu-package-architecture/).\n\nWhen a model reasons through intricate mathematical proofs, trajectories stretch across 8,000 to 16,000 tokens. Training a Critic to accurately predict the expected discounted return at every intermediate token is mathematically fragile. Critic errors amplify gradient variance, causing loss values to explode into NaNs.\n\n**GRPO (Group Relative Policy Optimization)** was pioneered by DeepSeek in the DeepSeekMath paper and scaled in DeepSeek-R1.\n\nIts core thesis is remarkably elegant: **Eliminate the Critic network entirely, sample a group of completions for each prompt, and use the group's empirical distribution as the baseline.**\n\nFor any input query $q$, the policy $\\pi_{\\theta_{old}}$ generates a group of $G$ distinct candidate completions:\n\n$${o_1, o_2, \\dots, o_G} \\sim \\pi_{\\theta_{old}}(q)$$\n\nThe verification environment (e.g., a regex answer parser or a compiler test runner) assigns scalar rewards to each completion:\n\n$${r_1, r_2, \\dots, r_G}$$\n\nRather than evaluating an absolute value network $V(s)$, GRPO computes the relative advantage $A_i$ of completion $o_i$ normalized against its peers:\n\n$$A_i = \\frac{r_i - \\text{mean}({r_1, \\dots, r_G})}{\\text{std}({r_1, \\dots, r_G}) + \\epsilon}$$\n\nRetaining the clipped surrogate mechanism from PPO, GRPO optimizes the following objective:\n\n$$\\mathcal{J}*{GRPO}(\\theta) = \\mathbb{E}*{q \\sim P(Q), {o_i}*{i=1}^G \\sim \\pi*{\\theta_{old}}(q)} \\left[ \\frac{1}{G} \\sum_{i=1}^{G} \\frac{1}{|o_i|} \\sum_{t=1}^{|o_i|} \\left( \\min \\left( \\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i,<t})}{\\pi_{\\theta_{old}}(o_{i,t} \\mid q, o_{i,<t})} A_{i,t}, \\; \\text{clip}\\left(\\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i,<t})}{\\pi_{\\theta_{old}}(o_{i,t} \\mid q, o_{i,<t})}, 1-\\epsilon, 1+\\epsilon\\right) A_{i,t} \\right) - \\beta D_{KL}(\\pi_\\theta \\parallel \\pi_{ref}) \\right) \\right]$$\n\nwhere the per-token KL divergence approximation is computed directly:\n\n$$D_{KL} = \\frac{\\pi_{ref}(o_{i,t} \\mid \\cdot)}{\\pi_\\theta(o_{i,t} \\mid \\cdot)} - \\log \\frac{\\pi_{ref}(o_{i,t} \\mid \\cdot)}{\\pi_\\theta(o_{i,t} \\mid \\cdot)} - 1$$\n\nThis architectural shift achieves two immediate advantages:\n\nIn the DeepSeek-R1-Zero experiment, researchers demonstrated that **without a single human SFT demonstration**, training a pure base model with GRPO induced complex reasoning behaviors:\n\nInstead of relying on a learned neural reward model (which often suffers from reward gaming), the environment enforces strict rule-based verifiers:\n\n`<think>` and `</think>` tags, with the final answer inside `<answer>`.\nAfter several thousand iterations, as trajectory length expanded, logs revealed models discovering self-correction unprompted:\n\n```\nUser: Is 9.11 larger than 9.9?\nAssistant:\n<think>\nFirst, compare 9.11 and 9.9.\nThe integer part is both 9.\nNow compare the tenths place: 9.11 has 1 in the tenths place, while 9.9 has 9 in the tenths place.\nWait! Let me rethink this... Am I confusing decimal places with software version numbers?\nIn software engineering, v9.11 is newer than v9.9, but mathematically, 9.9 = 9.90.\nSince 9.90 > 9.11, 9.9 is strictly larger.\nLet me double check the calculation: 9.9 - 9.11 = 0.79 > 0. Yes, absolutely correct.\n</think>\n<answer>\nNo, 9.9 is larger than 9.11.\n</answer>\n```\n\nFrom a reinforcement learning perspective, exploratory paths that verified intermediate results achieved a higher pass rate on difficult tasks than one-shot guesses. The group-relative advantage mechanism amplified these self-questioning trajectories.\n\nUsing Hugging Face's **TRL (Transformer Reinforcement Learning)** library, here is an end-to-end runnable script training a lightweight base model (such as `Qwen/Qwen2.5-1.5B-Instruct`) with GRPO:\n\n```\npip install torch transformers trl peft datasets accelerate\npython\nimport re\nimport torch\nfrom datasets import Dataset\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nfrom trl import GRPOTrainer, GRPOConfig\n\n# 1. Prepare deterministic verification dataset\ntrain_data = [\n    {\n        \"prompt\": \"Solve this equation: 3 * x + 7 = 22. What is x? Present your reasoning inside <think> and final value in <answer>.\",\n        \"target\": \"5\"\n    },\n    {\n        \"prompt\": \"A train travels 180 km in 3 hours. What is its speed in km/h? Think first in <think>, give value in <answer>.\",\n        \"target\": \"60\"\n    },\n    {\n        \"prompt\": \"If a square has an area of 64 cm^2, what is its perimeter in cm? Reason in <think>, answer in <answer>.\",\n        \"target\": \"32\"\n    }\n] * 100  # Expand dataset scale\n\ndataset = Dataset.from_list(train_data)\n\n# 2. Define deterministic rule-based reward functions\ndef correctness_reward_func(prompts, completions, target, **kwargs):\n    \"\"\"Verify if the content in <answer> strictly matches ground truth.\"\"\"\n    rewards = []\n    for completion, true_target in zip(completions, target):\n        match = re.search(r\"<answer>(.*?)</answer>\", completion, re.DOTALL)\n        if match:\n            pred = match.group(1).strip()\n            rewards.append(2.0 if pred == true_target.strip() else 0.0)\n        else:\n            rewards.append(0.0)\n    return rewards\n\ndef format_reward_func(completions, **kwargs):\n    \"\"\"Reward proper reasoning tag encapsulation.\"\"\"\n    rewards = []\n    pattern = r\"^<think>.*?</think>\\s*<answer>.*?</answer>$\"\n    for completion in completions:\n        if re.search(pattern, completion.strip(), re.DOTALL):\n            rewards.append(0.5)\n        else:\n            rewards.append(0.0)\n    return rewards\n\n# 3. Load model and tokenizer\nmodel_id = \"Qwen/Qwen2.5-1.5B-Instruct\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nif tokenizer.pad_token is None:\n    tokenizer.pad_token = tokenizer.eos_token\n\n# 4. Configure GRPO Hyperparameters\ntraining_args = GRPOConfig(\n    output_dir=\"./grpo_output_qwen\",\n    learning_rate=2e-5,\n    per_device_train_batch_size=2,\n    gradient_accumulation_steps=4,\n    num_generations=4,          # Group size G=4\n    max_prompt_length=256,\n    max_completion_length=1024,  # Ample space for CoT exploration\n    temperature=0.7,\n    warmup_ratio=0.1,\n    logging_steps=10,\n    max_steps=100,\n    save_strategy=\"steps\",\n    save_steps=50,\n    bf16=True,\n    report_to=\"none\"\n)\n\n# 5. Launch the Critic-Free Trainer\ntrainer = GRPOTrainer(\n    model=model_id,\n    reward_funcs=[correctness_reward_func, format_reward_func],\n    args=training_args,\n    train_dataset=dataset,\n)\n\nprint(\"🚀 Launching critic-free GRPO reinforcement learning pipeline...\")\ntrainer.train()\n```\n\n*For foundational training workflows and adapter memory tuning, review our [Comprehensive LLM Fine-Tuning Guide](https://blog.llmgo.top/en/articles/fine-tuning-guide/).*\n\nDeploying reasoning models in production requires addressing these operational considerations:\n\n`max_thinking_tokens`). For high-throughput infrastructure setup, refer to our GRPO replaces the parametric state-value estimation of the Bellman equation with empirical Monte Carlo group sampling. By generating a group of $G$ responses for the same prompt, the group mean serves as a dynamic, unbiased baseline. As long as the group size is sufficient ($G \\ge 4 \\sim 8$), the normalized advantage $\\frac{r_i - \\mu}{\\sigma}$ accurately signals relative trajectory quality.\n\nWhile DeepSeek-R1-Zero proved that cold-start reasoning can emerge from pure RL, practical production workflows benefit significantly from a lightweight initial SFT phase. Pure RL on raw base models frequently generates multilingual gibberish, formatting anomalies, and infinite repetition early in training. Starting with a few thousand curated chain-of-thought demonstrations accelerates convergence by over 5x while preserving readability.\n\nDPO (Direct Preference Optimization) is an offline supervised preference algorithm operating on static pairs of chosen and rejected responses $(y_w, y_l)$. It cannot discover novel reasoning pathways absent from the static dataset. GRPO is an active, online reinforcement learning algorithm where the model generates real-time samples evaluated dynamically by verifiable environment rewards, enabling open-ended exploration and spontaneous self-correction.\n\n*Originally published at [Nobita Talks AI](https://blog.llmgo.top/en/articles/test-time-compute-grpo/) on blog.llmgo.top.*", "url": "https://wpnews.pro/news/test-time-compute-and-grpo-in-practice-from-ppo-to-critic-free-reinforcement", "canonical_source": "https://dev.to/ifnodoraemon/test-time-compute-and-grpo-in-practice-from-ppo-to-critic-free-reinforcement-learning-4gpn", "published_at": "2026-09-20 03:13:28+00:00", "updated_at": "2026-09-20 03:54:38.240080+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "artificial-intelligence", "ai-infrastructure"], "entities": ["DeepSeek", "OpenAI", "DeepSeek-R1", "OpenAI o1", "OpenAI o3", "NVIDIA", "GRPO", "PPO"], "alternates": {"html": "https://wpnews.pro/news/test-time-compute-and-grpo-in-practice-from-ppo-to-critic-free-reinforcement", "markdown": "https://wpnews.pro/news/test-time-compute-and-grpo-in-practice-from-ppo-to-critic-free-reinforcement.md", "text": "https://wpnews.pro/news/test-time-compute-and-grpo-in-practice-from-ppo-to-critic-free-reinforcement.txt", "jsonld": "https://wpnews.pro/news/test-time-compute-and-grpo-in-practice-from-ppo-to-critic-free-reinforcement.jsonld"}}