When the answer can be checked: Training a model with GRPO and GDPO on a security task Equixly AI Research (EAIR) reports that training an open-weight model with eight variants of GRPO and GDPO on a 195-question API and application security benchmark raised test accuracy from a 52.3% base to between 59.3% and 65.0% after 80 training steps, a gain of 7.0 to 12.6 points established for seven of the eight variants. None of the planned comparisons between variants found an established difference, since with 195 test questions and three seeds differences below about four points cannot be resolved, and with reward weights of 1, ½ and ¼ GRPO and GDPO both reached 63.1% accuracy. The researchers also found that roughly half of answer groups carried no learning signal at the start of training and 60–70% by the end, and that the weights assigned to multiple rewarded goals determine what the model prefers. Four training methods, one security task, two lessons about rewards. By Equixly AI Research EAIR Abstract. This post clarifies the role of four training methods for language models. DPO Direct Preference Optimization aligns a model with preferences expressed as pairs of answers; PPO Proximal Policy Optimization , GRPO Group Relative Policy Optimization , and GDPO Group reward-Decoupled Normalization Policy Optimization improve a skill whenever a program can check the answer. We explain each method, then report a showcase experiment on a security task from Equixly’s internal benchmarks: multiple-choice questions on difficult exploitation scenarios in API and application security, where the answer can be checked against a key. We trained an open-weight model with eight variants of GRPO and GDPO, three seeds each, under a protocol fixed before the first run. Every variant raised test accuracy by 7 to 13 points over the base model, and none was reliably better than another. Two observations explain most of what we saw: on this task, most groups of answers carried no learning signal, and when several goals are rewarded, the weights given to them decide what the model prefers. These results come from one task and one training budget; we state them as observations about this showcase, not as general rules. 1 Key takeaways Both GRPO and GDPO improved the model. The base model answered 52.3% of the 195 test questions correctly. After 80 training steps, the eight variants reached 59.3% to 65.0%, a gain of 7.0 to 12.6 points, established for seven of them Table 5 . The variants were not distinguishable from each other. None of the planned comparisons between variants found an established difference. With 195 test questions and three seeds, differences below about four points cannot be resolved. When several goals are rewarded, choose the weights deliberately. Our reward had three goals: a correct letter, the requested two-line format, and a short reason. With equal weights, a correct but long answer and a wrong but short one can receive the same reward. Weights of 1, ½, and ¼ make every combination of goals score differently. With these weights, GRPO and GDPO reached the same accuracy, 63.1%. Most groups of answers taught nothing about correctness. GRPO learns by comparing several answers to the same question. When all of them are right, or all wrong, there is nothing to compare. This was the case for about half of the groups at the start of training and 60–70% at the end, and the total reward did not show it. Table 1 turns these points, and the background of Sec. 3 3-the-methods , into a first choice of method. Table 1. Which method fits which kind of feedback. 2 Introduction Equixly runs continuous offensive security testing for APIs and applications https://equixly.com/platform/ . Its agents map an attack surface, chain requests into exploit paths, and prove each finding from inside the CI/CD pipeline. The agents rely on a model that Equixly trains in-house. Part of that work is deciding how to use feedback, such as a verdict from a target or a judgment that one answer is better than another, to improve the model. The four methods in this post come from two lines of work with different goals. - Preference alignment. DPO 1 trains a model on pairs of answers in which one is marked as better. The theory behind it comes from RLHF Reinforcement Learning from Human Feedback 2 – 5 , the line of work that made language models follow instructions. We use RLHF only as the background needed to explain DPO. - Reinforcement learning with a verifiable reward. When a program can check the answer, the check itself is the reward. The model generates answers, the checker scores them, and the model is pushed toward the answers that scored well. PPO 6 was the first method widely used for this with language models. GRPO 7 , 8 removes PPO’s critic network, and GDPO 9 changes how GRPO combines several rewards. Most published comparisons of GRPO and GDPO use mathematics problems. We wanted to see how they behave on a security task. Our task is a set of multiple-choice questions on exploitation scenarios with an answer key Sec. 4.1 41-the-task , so it belongs to the second line of work, and the experiment compares eight variants of GRPO and GDPO Table 4 . We describe PPO and DPO for context. Sec. 4.4 44-why-ppo-and-dpo-are-not-in-the-experiment explains why neither is part of the experiment. 3 The methods 3.1 From a score to an advantage All four methods need to know whether an answer was better or worse than what the model usually produces for the same question. A raw score does not say this: a score of 0.7 may be poor on an easy question and excellent on a hard one. The quantity the methods use is the advantage : the score minus a baseline for that question. A positive advantage makes the answer more likely, a negative one less likely. The methods differ mainly in how they estimate the baseline. 3.2 PPO PPO 6 trains two networks: the language model itself, called the policy , and a critic or value model that predicts the reward expected from each point of a partially written answer. The advantage of each token is computed with GAE Generalized Advantage Estimation 10 . GAE combines the critic’s predictions at successive tokens with the final reward, and a parameter λ sets how much it relies on the critic’s predictions low λ or on the actual outcome λ = 1 . The result is one advantage per token. PPO also limits how far one update can move the model. It compares the probability of each action under the current policy with its probability under the policy that generated the data and clips the ratio to a band around 1. The objective, Eq. 7 of the PPO paper, is where r