Follow-up: Part 3 scales the CISPO + REPO-R recipe to Qwen3.8-27B and 600 steps, with a one-change-per-run holdout ladder and an advantage-floor failure we found in a harsher environment.
Part 1 analyzed how PPO, GRPO, DAPO, and GDPO balance competing reward objectives in theory. In this follow-up empirical benchmark, we test the practical question engineering teams actually face: which trainer algorithm actually produces a model that solves brand-new, unseen problems?
We ran seven distinct RL algorithms on Qwen3-14B in our deterministic decentralized exchange (DEX) arbitrage gym across two reasoning regimes: No-think (direct execution) and Thinking-2048 (with chain-of-thought tokens enabled). The results revealed a severe cautionary tale for post-training teams.
The Generalization Trap: In thinking mode, CISPO achieved the highest training reward curve (0.8951 last-10 mean), yet its performance on frozen test tasks collapsed to 0.6068. Meanwhile, the raw Qwen3-14B base model—with thinking enabled and zero RL training—scored 0.6830. If you only look at your training curves, you will celebrate a run that is actively degrading your model.
1. What the agent does: The DEX gym #
DEX stands for decentralized exchange. In our gym, the agent is presented with a frozen snapshot of automated market maker (AMM) liquidity pools: token reserves, exchange fees, gas prices, and routing constraints. The agent’s task is to find an optimal two-to-four pool arbitrage cycle and compute the exact input amount (amount_in_wei) that returns more of the starting asset after all fees and gas deductions.
WETH (Wrapped Ether) is the asset at both ends of the route. Consider an intuitive triangular arbitrage cycle: the model starts with WETH, swaps into USDC, swaps USDC into UNI, and swaps UNI back into WETH.
- If the trade size is too small: The fixed Ethereum gas fees eat 100% of the price discrepancy, yielding a negative net return.
- If the trade size is too large: The constant-product liquidity formula () causes severe price slippage, collapsing the margin.
- If the pool sequence is invalid: The simulated EVM contract reverts and the transaction fails immediately.
The agent interacts using structured tool calls: inspecting pool reserves, proposing a pool route and trade amount, evaluating feedback, and finalizing. The verifier replays the transaction bytecode using a native Foundry / revm EVM execution engine. It scores the verified mathematical execution on the simulated blockchain, rather than trusting any self-reported text output by the model.
The composite training reward consists of three distinct channels:
- Native Task Score (R_native ∈ [0, 1]): The verified net profit margin returned to the starting wallet after all pool fees and gas costs.
- **Execution Efficiency (R_efficiency ∈ [0, 1]):** Rewards solving the task compactly in 2–3 tool turns instead of exhausting all 6 turns:`max(0, 1 − tool_steps / 6)` .
- **Reliability Gate (R_reliability ∈ {-1, +1}):** +1.0 if the transaction executed cleanly without syntax errors or contract reverts; −1.0 if the call failed.
2. What we held fixed: The training contract #
To isolate the impact of the RL trainer algorithms, every other variable was strictly controlled:
- Model and Quantization: Qwen3-14B (revision
40c06982…), quantized NF4 weights with BF16 compute, LoRA rank 32, alpha 64. - Compute Hardware: One dedicated NVIDIA RTX PRO 6000 Blackwell 96 GB per training run, Hugging Face generation, fused SDPA attention, CUDA graphs disabled.
- Optimizer Schedule: Fixed seed 42, learning rate 1e-5 with linear decay, 50 optimizer steps, group size 4, eight completions per fresh rollout batch, two policy updates per rollout. Curriculum learning disabled.
- **Reasoning Modes:**
- **No-think:** Chain-of-thought thinking disabled; max 1,024 generated tokens per tool turn (max 8,192 trajectory cap).
- Thinking-2048: Model allocated up to 2,048 thinking tokens inside a 3,072-token turn budget (max 22,528 trajectory cap).
- Frozen Holdout Evaluation: Evaluated on the final step-50 checkpoint. 5 task families × 2 held-out seeds × 4 samples = 40 episodes across10 distinct unseen scenarios . Untrained starting checkpoints evaluated identically in both reasoning modes.
3. Trainer features: Seven algorithm bundles #
In modern post-training, an RL algorithm is not a monolithic block. It is a bundle of three distinct modular decisions: which rollouts enter the training batch, how multiple rewards are combined into policy advantages, and how that advantage scales policy gradient updates.
| Architectural choices across the seven benchmarked trainer configurations. PPO was excluded due to its separate Critic VRAM requirement. | ||||
|---|---|---|---|---|
| Configuration | Loss / averaging | Reward → advantage | Dynamic sampling | Control |
| --- | --- | --- | --- | --- |
| GRPO | Clipped surrogate; sequence-level averaging | Joint reward → group normalization | Off | Monitor only |
| DAPO | Clipped surrogate; token-level averaging | Joint reward → group normalization | Off | Fixed asymmetric clip: 0.20 / 0.28 |
| DAPO-refill | Same DAPO loss | Same as DAPO | On; at most 2 refill rounds | Same fixed clip |
| GDPO | DAPO loss in this implementation | Normalize each reward, then combine | Off | Fixed asymmetric clip |
| CISPO | Detached, capped importance weights | Joint reward → group normalization | Off | Importance-weight cap 1.20 |
| DAPO + ADAPO | DAPO loss | Same as DAPO | Off | Entropy feedback adjusts upper clip |
| DAPO + REPO-R | DAPO loss | Token-level advantage shaping | Off | Entropy feedback controls ζ |
| CISPO + REPO-R w5 | CISPO loss; cap stays 1.20 | Token-level advantage shaping | Off / separate refill arm | 5-step target window; ζ ≥ 0 |
GDPO decouples reward-channel normalization before combining advantages. CISPO replaces ratio clipping with direct importance weight clipping (capped at 1.20). ADAPO and REPO-R introduce adaptive entropy control: ADAPO adjusts clipping bounds based on policy entropy, while REPO-R shapes token-level advantages. DAPO-refill activates replacement sampling when a rollout group exhibits zero reward variance, preventing zero-gradient wasted batches.
4. Full results: Train vs. holdout test #
The CISPO Inversion Trap
Train: 0.8951 → Test: 0.6068 CISPO had the highest training reward but the lowest holdout score. One likely reading: it kept raising the composite training reward without improving the native score on tasks it had not seen.
Thinking Baseline Beats RL
Raw base: 0.6830 (37/40) Untrained Qwen3-14B with 2,048 thinking tokens beat 6 of the 7 trained models. Always benchmark against an untrained thinking baseline.
DAPO Leads Generalization
No-think: 0.6467 · Think: 0.7105 DAPO delivered the highest native holdout score in no-think mode (0.6467, 39/40), and DAPO-refill led thinking mode (0.7105, 38/40).
The 5× Compute Tax
13.5 hrs vs 2.6 hrs Thinking steps took 4.5–5.9× longer wall-clock time. You must balance the evaluation gain (+0.0275) against a 5× compute budget increase.
No-think mode: Every trained model beats the starting baseline
| No-think mode · snapshot September 14, 2026. “Train last-10” averages the final 10 fresh rollout rewards. “Holdout native” is the strict verifier score on unseen test scenarios. | |||||||
|---|---|---|---|---|---|---|---|
| Configuration | Steps | Train mean | Train last-10 | Holdout native | Successes | Step, s | Step-hours |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Starting modelbaseline | 0 | — | — | 0.3856 | 16/40 | — | — |
| GRPO | 50/50 | 0.7038 | 0.7729 | 0.6424 | 38/40 | 177 | 2.46 |
| DAPOtop holdout | 50/50 | 0.6519 | 0.7142 | 0.6467 | 39/40 | 176 | 2.45 |
| DAPO-refill | 50/50 | 0.6564 | 0.7585 | 0.5839 | 34/40 | 224 | 3.11 |
| GDPO | 50/50 | 0.6488 | 0.7727 | 0.6245 | 36/40 | 167 | 2.32 |
| CISPO | 50/50 | 0.6486 | 0.7486 | 0.6190 | 36/40 | 164 | 2.27 |
| DAPO + ADAPO | 50/50 | 0.6694 | 0.7525 | 0.6332 | 37/40 | 168 | 2.33 |
| DAPO + REPO-R | 50/50 | 0.6373 | 0.7057 | 0.5724 | 31/40 | 156 | 2.16 |
Without reasoning tokens, post-training delivers an unambiguous leap forward. The untrained base model scored only 0.3856 (passing 16/40 episodes). DAPO achieved 0.6467 (39/40 successes), closely followed by GRPO at 0.6424. Every single trained configuration substantially outperformed the starting model.
Thinking-2048 mode: The starting model sets a high bar
| Thinking-2048 mode · snapshot September 14, 2026. “Train last-10” averages the final 10 fresh rollout rewards. “Holdout native” is the strict verifier score on unseen test scenarios. | |||||||
|---|---|---|---|---|---|---|---|
| Configuration | Steps | Train mean | Train last-10 | Holdout native | Successes | Step, s | Step-hours |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Starting modelbaseline | 0 | — | — | 0.6830 | 37/40 | — | — |
| GRPO | 50/50 | 0.8362 | 0.8824 | 0.6798 | 38/40 | 929 | 12.90 |
| DAPO | 50/50 | 0.8036 | 0.7916 | 0.6703 | 38/40 | 938 | 13.02 |
| DAPO-refilltop holdout | 50/50 | 0.8275 | 0.8564 | 0.7105 | 38/40 | 1016 | 14.10 |
| GDPO | 50/50 | 0.8190 | 0.8253 | 0.6602 | 35/40 | 938 | 13.03 |
| CISPOdiverged | 50/50 | 0.8388 | 0.8951↑ | 0.6068 | 33/40 | 907 | 12.60 |
| DAPO + ADAPO | 50/50 | 0.8185 | 0.8815 | 0.6250 | 34/40 | 887 | 12.32 |
| DAPO + REPO-R | 50/50 | 0.8285 | 0.8206 | 0.6461 | 34/40 | 922 | 12.81 |
| CISPO + REPO-R w5 | 50/50 | 0.8165 | 0.8332 | 0.6243 | 34/40 | 937 | 13.01 |
| CISPO + REPO-R w5 + refill partial | 41/50 | 0.7908 | 0.8302 | — | — | 912 | 10.39 |
When chain-of-thought thinking tokens are enabled, the starting model achieves 0.6830 straight out of the box (passing 37/40 episodes). Only DAPO-refill surpassed the untrained model on native holdout reward (0.7105, an incremental gain of +0.0275). GRPO (0.6798) and standard DAPO (0.6703) finished slightly below the base model’s score despite high success rates (38/40), demonstrating that success counts and solution quality answer different questions.
5. Explore the curves: Rollout telemetry #
Each 50-step run generated 25 fresh rollout reward checkpoints (even optimizer steps update on rollouts generated on odd steps). The chart below shows three of them in thinking mode:
Use the interactive explorer below to compare any two trainer configurations across the 50-step trajectory:
| Step | CISPO | CISPO + REPO-R w5 |
|---|---|---|
| 1 | 0.708017 | 0.708017 |
| 3 | 0.953178 | 0.685857 |
| 5 | 0.842710 | 0.740580 |
| 7 | 0.833925 | 0.708557 |
| 9 | 0.716074 | 0.756640 |
| 11 | 0.787146 | 0.897562 |
| 13 | 0.720173 | 0.812017 |
| 15 | 0.962763 | 0.807996 |
| 17 | 0.699492 | 0.848704 |
| 19 | 0.883526 | 1.009395 |
| 21 | 0.776956 | 0.928267 |
| 23 | 1.013888 | 1.006660 |
| 25 | 0.555751 | 0.658180 |
| 27 | 0.765734 | 0.781008 |
| 29 | 0.798723 | 0.731481 |
| 31 | 1.093313 | 0.989723 |
| 33 | 0.884916 | 0.947463 |
| 35 | 0.870928 | 0.748010 |
| 37 | 0.937840 | 0.551544 |
| 39 | 0.973666 | 0.685541 |
| 41 | 0.800041 | 0.905815 |
| 43 | 0.892790 | 0.938511 |
| 45 | 0.981199 | 0.782173 |
| 47 | 0.566612 | 0.881300 |
| 49 | 0.949583 | 0.901488 |
Over its last ten reward points CISPO averaged 0.895 in thinking mode and DAPO 0.792, yet on unseen tasks DAPO scored 0.670 and CISPO 0.607. The training reward is measured on training tasks and includes bonuses, so it is no substitute for a frozen holdout.
6. Time and memory: The cost of thinking #
Test-time reasoning comes with a heavy computational invoice. Across all seven algorithms, an average thinking step took 4.5–5.9× longer than a no-think step. Fifty training steps required approximately 12.3–14.1 hours of GPU execution in thinking mode, compared to 2.2–3.1 hours in no-think mode.
| Peak VRAM consumption during thinking-mode training on NVIDIA RTX PRO 6000 Blackwell 96 GB. Memory measured in GiB (binary). | ||
|---|---|---|
| Configuration | Peak allocated, GiB | Peak reserved, GiB |
| --- | --- | --- |
| GRPO | 69.1 | 76.9 |
| DAPO | 66.5 | 74.0 |
| DAPO-refill | 68.5 | 76.3 |
| GDPO | 68.3 | 76.1 |
| CISPO | 70.0 | 77.8 |
| DAPO + ADAPO | 71.8 | 79.6 |
| DAPO + REPO-R | 68.0 | 75.5 |
| CISPO + REPO-R w5 | 66.9 | 74.4 |
| CISPO + REPO-R w5 + refill (partial) | 70.3 | 77.8 | In thinking mode, peak memory stayed between 66.5 and 71.8 GiB allocated (74.0–79.6 GiB reserved), within the 96 GB budget of the Blackwell GPU. The primary operational bottleneck is wall-clock rollout latency, not GPU memory.
7. Combining features: The CISPO + REPO-R hybrid #
Can we rescue CISPO’s optimization speed while fixing its generalization collapse? We tested a hybrid architecture: using CISPO’s capped importance weights (cap 1.20) combined with REPO-R’s token-level advantage shaping.
To stabilize entropy dynamics, we introduced a nonnegative window-5 controller: the first 5 optimizer steps record entropy drift without intervention (ζ = 0). Once calibrated, the entropy control strength is constrained strictly to 0 ≤ ζ ≤ 0.05.
The completed hybrid achieved 0.6243 native holdout reward and 34/40 successes, outperforming standalone CISPO (0.6068, 33/40). While this +0.0175 lift confirms that advantage shaping mitigates policy collapse, the hybrid still trailed the raw untrained thinking baseline (0.6830).
Enabled is not the same as exercised
In no-think mode, DAPO-refill triggered 19 replacement rollout rounds. In thinking mode, DAPO-refill triggered zero refills—the model never produced an all-identical reward batch. Therefore, we cannot attribute the thinking DAPO-refill lead (0.7105) to the refill mechanism itself; it functioned identically to standard DAPO under that seed.
8. What to test next: Next iteration & replication #
Based on these findings, our production post-training recommendations for multi-step reasoning workflows are:
- In No-Think Mode: Standard DAPO and GRPO are clear, cost-effective winners. They raised the holdout score from 0.386 to about 0.64–0.65 (16/40 to 38–39/40 successes) in about 2.5 GPU-hours.
- In Reasoning Mode: Always evaluate your untrained base model with reasoning enabled before declaring training success. DAPO-refill is the primary candidate for replication.
- Never Pick a Checkpoint by Training Curves: CISPO proved that soaring training curves can mask test-set degradation. All promotion decisions must be gated by frozen holdout benchmarks.
- Multi-Seed Replication: Expand seed sweeps (seeds 43, 44) across novel scenario families to confirm that DAPO-refill’s +0.0275 edge holds across market distributions.
Evidence and method references
All raw empirical telemetry is publicly available. The public JSON export contains complete run manifests, SHA256 hashes, all 791 recorded optimizer steps, 396 fresh reward logs, and 680 evaluated holdout episodes. The Markdown companion provides searchable per-step records.
Foundational algorithm publications: DeepSeekMath / GRPO; DAPO; GDPO; MiniMax-M1 / CISPO; ADAPO and REPO-R.