cd /news/ai-research/when-the-answer-can-be-checked-train… · home › topics › ai-research › article
[ARTICLE · art-171163] src=equixly.com ↗ pub= topic=ai-research verified=true sentiment=· neutral

When the answer can be checked: Training a model with GRPO and GDPO on a security task

Equixly AI Research (EAIR) reports that training an open-weight model with eight variants of GRPO and GDPO on a 195-question API and application security benchmark raised test accuracy from a 52.3% base to between 59.3% and 65.0% after 80 training steps, a gain of 7.0 to 12.6 points established for seven of the eight variants. None of the planned comparisons between variants found an established difference, since with 195 test questions and three seeds differences below about four points cannot be resolved, and with reward weights of 1, ½ and ¼ GRPO and GDPO both reached 63.1% accuracy. The researchers also found that roughly half of answer groups carried no learning signal at the start of training and 60–70% by the end, and that the weights assigned to multiple rewarded goals determine what the model prefers.

by read28 min views1 publishedNov 17, 2026
When the answer can be checked: Training a model with GRPO and GDPO on a security task
Image: Equixly (auto-discovered)

Four training methods, one security task, two lessons about rewards.

By Equixly AI Research (EAIR) Abstract. This post clarifies the role of four training methods for language models. DPO (Direct Preference Optimization) aligns a model with preferences expressed as pairs of answers; PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), and GDPO (Group reward-Decoupled Normalization Policy Optimization) improve a skill whenever a program can check the answer. We explain each method, then report a showcase experiment on a security task from Equixly’s internal benchmarks: multiple-choice questions on difficult exploitation scenarios in API and application security, where the answer can be checked against a key. We trained an open-weight model with eight variants of GRPO and GDPO, three seeds each, under a protocol fixed before the first run. Every variant raised test accuracy by 7 to 13 points over the base model, and none was reliably better than another. Two observations explain most of what we saw: on this task, most groups of answers carried no learning signal, and when several goals are rewarded, the weights given to them decide what the model prefers. These results come from one task and one training budget; we state them as observations about this showcase, not as general rules.

1 Key takeaways #

Both GRPO and GDPO improved the model. The base model answered 52.3% of the 195 test questions correctly. After 80 training steps, the eight variants reached 59.3% to 65.0%, a gain of 7.0 to 12.6 points, established for seven of them (Table 5).

The variants were not distinguishable from each other. None of the planned comparisons between variants found an established difference. With 195 test questions and three seeds, differences below about four points cannot be resolved.

When several goals are rewarded, choose the weights deliberately. Our reward had three goals: a correct letter, the requested two-line format, and a short reason. With equal weights, a correct but long answer and a wrong but short one can receive the same reward. Weights of 1, ½, and ¼ make every combination of goals score differently. With these weights, GRPO and GDPO reached the same accuracy, 63.1%.

Most groups of answers taught nothing about correctness. GRPO learns by comparing several answers to the same question. When all of them are right, or all wrong, there is nothing to compare. This was the case for about half of the groups at the start of training and 60–70% at the end, and the total reward did not show it.

Table 1 turns these points, and the background of Sec. 3, into a first choice of method. Table 1. Which method fits which kind of feedback.

2 Introduction #

Equixly runs continuous offensive security testing for APIs and applications. Its agents map an attack surface, chain requests into exploit paths, and prove each finding from inside the CI/CD pipeline. The agents rely on a model that Equixly trains in-house. Part of that work is deciding how to use feedback, such as a verdict from a target or a judgment that one answer is better than another, to improve the model.

The four methods in this post come from two lines of work with different goals.

  • Preference alignment. DPO [1] trains a model on pairs of answers in which one is marked as better. The theory behind it comes from RLHF (Reinforcement Learning from Human Feedback) [2]–[5], the line of work that made language models follow instructions. We use RLHF only as the background needed to explain DPO.
  • Reinforcement learning with a verifiable reward. When a program can check the answer, the check itself is the reward. The model generates answers, the checker scores them, and the model is pushed toward the answers that scored well. PPO [6] was the first method widely used for this with language models. GRPO [7], [8] removes PPO’s critic network, and GDPO [9] changes how GRPO combines several rewards.

Most published comparisons of GRPO and GDPO use mathematics problems. We wanted to see how they behave on a security task. Our task is a set of multiple-choice questions on exploitation scenarios with an answer key (Sec. 4.1), so it belongs to the second line of work, and the experiment compares eight variants of GRPO and GDPO (Table 4). We describe PPO and DPO for context. Sec. 4.4 explains why neither is part of the experiment.

3 The methods #

3.1 From a score to an advantage

All four methods need to know whether an answer was better or worse than what the model usually produces for the same question. A raw score does not say this: a score of 0.7 may be poor on an easy question and excellent on a hard one. The quantity the methods use is the advantage: the score minus a baseline for that question. A positive advantage makes the answer more likely, a negative one less likely. The methods differ mainly in how they estimate the baseline.

3.2 PPO

PPO [6] trains two networks: the language model itself, called the policy, and a critic (or value model) that predicts the reward expected from each point of a partially written answer. The advantage of each token is computed with GAE (Generalized Advantage Estimation) [10]. GAE combines the critic’s predictions at successive tokens with the final reward, and a parameter λ sets how much it relies on the critic’s predictions (low λ) or on the actual outcome (λ = 1). The result is one advantage per token.

PPO also limits how far one update can move the model. It compares the probability of each action under the current policy with its probability under the policy that generated the data and clips the ratio to a band around 1. The objective, Eq. (7) of the PPO paper, is

where r<sub>t</sub>(θ) is the probability ratio (not a reward), Â<sub>t</sub> is the advantage estimate, and ε is typically 0.2, which gives the band 0.8–1.2. For a language model, the state s<sub>t</sub> is the question together with the tokens written so far, (q, o<sub><t</sub>), and the action a<sub>t</sub> is the next token o<sub>t</sub>.

When PPO is used to fine-tune a language model, a second limit is usually added: a KL (Kullback–Leibler) penalty, subtracted from the reward token by token, that measures how far the model has drifted from a frozen copy of its starting point, the reference. Clipping limits each update; the KL penalty limits the total drift over the whole run. PPO therefore keeps three models in memory: the policy, the critic, and the reference.

3.3 DPO: Learning from preference pairs

DPO [1] learns from pairs of answers to the same prompt x in which one, y<sub>w</sub>, is marked as preferred and the other, y<sub>l</sub>, as rejected. The labels may come from people or from a judge model.

Its theory comes from RLHF [2]–[5]. There, a reward model is first trained on such comparisons, and the language model is then optimized against it with PPO and a KL penalty toward the reference. Rafailov et al. show that the optimum of that objective can be reached without either stage: under the Bradley–Terry model of preferences, the log-probability ratio between the trained model and the reference acts as an implicit reward, and training becomes a classification loss over the pairs. The loss, Eq. (7) of the DPO paper, is

where 𝒟 is the dataset of pairs, σ is the logistic function, and β sets how strongly the model is held to the reference. DPO needs only two models, the trained one and the reference, and one parameter, β. It assumes the labels are reliable: in every pair, y<sub>w</sub> really is the better answer. In its original form, the pairs are collected once, before training, so the data do not change as the model improves. Iterative and online variants [11], [12] address this by sampling new pairs from the current model during training.

3.4 GRPO: A group of answers instead of a critic

When answers can be checked automatically, the model can generate many of them. GRPO [7] generates a group of G answers o<sub>1</sub>, …, o<sub>G</sub> to the same question and uses the group itself as the baseline (RLOO, REINFORCE Leave-One-Out [13], does the same with the mean of the other G − 1 answers). With one reward r<sub>i</sub> per answer, the rewards of the group, r = {r<sub>1</sub>, …, r<sub>G</sub>}, are normalized by their mean and standard deviation, and every token of answer i receives the same advantage (Sec. 4.1.2 of [7]):

Animation 1. GRPO compares each answer with the mean and spread of its group: answers above the mean get a positive advantage, answers below it a negative one.

The objective, Eq. (3) of [7], is PPO’s clipped objective averaged over the tokens of each answer and over the G answers, with a KL penalty toward the reference added to the loss instead of to the reward:

where ρ<sub>i,t</sub> is the probability ratio of Eq. (1) for token t of answer i, the expectation is over questions and over groups sampled from π<sub>θ<sub>old</sub></sub>, and the KL term is estimated token by token with the paper’s Eq. (4).

Figure 1. PPO estimates a baseline for every token with a trained critic; GRPO uses the mean and spread of G answers to the same question. The reference model is needed only when the KL weight β is above zero.

Without a critic, there is one network fewer to train and to keep in memory. The cost is that every token of an answer receives the same advantage, so GRPO cannot tell which part of a long answer was responsible for the result. DeepSeek-R1 [8] showed that this trade works at scale on reasoning tasks.

GRPO has a specific weakness. If all G answers in a group receive the same reward, every advantage in the group is zero, and the question contributes nothing to the update. We call these silent groups. They occur when a question is too easy (all answers right) or too hard (all wrong). Two common remedies are larger groups, which make a unanimous group less likely, and the dynamic sampling of DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) [14], which generates extra groups and discards those whose answers are all right or all wrong until the batch is full.

Animation 2. A group with both right and wrong answers carries a learning signal; a group whose answers are all right or all wrong is silent. As training goes on, more groups fall silent.

Dr. GRPO [15] addresses a different issue. It removes the division by the group’s standard deviation in Eq. (3) and the division by the answer length |o<sub>i</sub>| in Eq. (4). The first gives extra weight to questions whose answers barely differ; the second changes how much short and long answers count. Dr. GRPO (GRPO Done Right) is a correction of these biases, not a remedy for silent groups. We include it as one of the variants.

3.5 GDPO: Normalizing each goal separately

Often, several goals are rewarded at once, such as a correct answer, a valid format, and a short explanation. GRPO adds the rewards for each goal into one number per answer and normalizes that total within the group with Eq. (3). GDPO [9] keeps the goals apart until the end. In the paper’s notation, i is the question; j the answer within its group of G; and k one of n goals, with r<sub>k</sub><sup>(i,j)</sup> the reward of goal k for that answer. GDPO proceeds in three steps.

Step 1: Normalize each goal within the group (Eq. (4) of [9]). Each goal’s reward is centered and scaled using only the G answers to the same question: This is Eq. (3) applied to one goal at a time. A goal that every answer in the group passes, or that every answer fails, has no spread and contributes zero for that group. A goal that only one answer passes gives that answer a large positive value and the others a small negative one. Each goal is therefore measured on its own scale, so a goal whose rewards happen to be larger, or more variable, cannot drown out the others.

Step 2: Add the normalized values (Eq. (5) of [9]). The per-goal values are summed into one number per answer: Because each term has been normalized within the group, the sum weighs the goals by how unusual the answer’s result is for each goal, not by the size of the raw rewards. This is the point where GDPO and GRPO differ: GRPO sums first and normalizes the total once; GDPO normalizes each goal first and then sums.

Step 3: Normalize over the batch (Eq. (6) of [9]). The summed value is centered and scaled once more, this time over all answers to all questions in the training batch:

where ε is a small constant that avoids division by zero. The authors add this step so that the scale of the final advantage stays stable and does not grow as more goals are added, and report that removing it occasionally leads to convergence failures.

The result, Â<sub>sum</sub><sup>(i,j)</sup>, is the advantage used in the objective of Eq. (4) in place of Â<sub>i,t</sub>. The paper also allows a weight per goal in the sum of step 2 (its Eq. (7)); Sec. 4.2 says how we used it.

4 The experiment #

The questions, metrics, comparisons, and decision rules were written down before the first training run. Three later changes were made without looking at test results and are reported with the results they produced: more training steps after a smoke test, more seeds, and one exploratory variant with a larger learning rate (Table 4).

4.1 The task

The task comes from Equixly’s internal benchmark suite: a set of multiple-choice questions on difficult exploitation scenarios in API and application security. The set is not public, so we describe it here.

Each question describes a web service: what it exposes and how it responds to requests. It then lists four testing actions an attacker could try next, such as a specific injection type, an authorization check, or a parameter manipulation. Exactly one action fits the behavior described; the model must name it by letter.

The task therefore measures whether the model picks the right test for the situation. It does not measure whether the model can carry the test out: no service is contacted, and the reward only compares the letter with the answer key.

The set has 496 questions across many vulnerability categories, with the correct letter balanced across A–D. We prepared it in three ways:

  • No giveaways. In 195 of the 496 questions, the scenario named the vulnerability class. We replaced the name with a neutral identifier, so the model has to infer the class from the service’s behavior. On these questions, the base model answers 33.8% correctly, against 63.6% on the others. Combined in their proportions, the two figures give its overall accuracy of about 52%.
  • A clean split. Questions are split 60/40 by scenario and category, 301 for training and 195 for testing, with no near-duplicates across the split. At the time of writing, 20 of the test questions are under revision.Sec. 5.2 checks that they do not change any conclusion.
  • A transfer set. 239 exam-style security questions from a different source show whether training on the task damages other knowledge.

The prompt uses the model’s chat template with its thinking mode disabled. It asks for exactly two lines, Reason: <one sentence> and Answer: <letter>, and is the same in training and evaluation.

Before training, we sampled 16 answers per training question from the base model and checked four things that GRPO-style training needs (Table 2).

Table 2. Four checks on the base model before training, from 16 sampled answers per training question.

The same samples give each training question a difficulty: easy if at least 75% of its 16 samples are correct (112 questions); hard if at most 25% are (134 questions); and medium in between (55 questions). Sec. 6.2 uses this ranking.

4.2 Rewards

Each answer is scored on three binary goals:

  • Correct : the letter after the lastAnswer: matches the key;

  • Format : the answer consists of exactly the two requested lines;

  • Short : the one-sentence reason has at most 23 words. The limit of 23 words is the 30th percentile of the base model’s reasons, fixed in advance, so that brevity starts as the rarest goal: 34% of base samples are short, 47% correct, and 99% well formatted.

The three goals are combined in one of two ways:

  1. With equal weights (1, 1, 1), the total reward is the number of goals met.
  2. With ranked weights (1, ½, ¼), each goal is worth more than all the less important goals together, so every combination of goals has its own total and a correct answer always scores above a wrong one.

Animation 3. With equal weights, a correct but long answer and a wrong but short one receive the same reward; with ranked weights (1, ½, ¼), the correct answer always scores higher.

Under GRPO, the weights multiply the rewards before the group normalization of Eq. (3). Under GDPO, they multiply the per-goal values in the sum of step 2 (Eq. (7) of [9]). The reward code is covered by 12 unit tests, which also check that our advantage computation matches the training library’s to within 10⁻⁵.

4.3 Training setup and variants

The trained model is an open-weight instruction model that we do not name here. A second, smaller model of the same family is used only for the full fine-tuning memory measurements in Sec. 6.3. Where the two must be told apart, we call them the larger model and the small model. Table 3 gives the configuration shared by all GRPO-family variants.

Table 3. Training configuration, shared by all GRPO-family variants.

The reference variant is GRPO with ranked weights. Every other variant changes one or two settings, listed in the second column of Table 4.

Table 4. The eight variants. Each has three seeds; variants with the same seed see the same questions in the same order.

Before evaluation, we checked that vLLM and the Hugging Face implementation produce identical answers (40 of 40) and that every trained adapter changes the model after .

Statistics. The seed is the unit of replication. A difference counts as established only if the 95% t-interval of its per-seed values excludes zero. Five comparisons between variants were planned before the first run (Table 6); the preregistered analysis also applied a Holm correction [18] to them, which changes nothing here because no interval excludes zero even without it.

With 195 test questions and three seeds, differences smaller than about four points cannot be resolved, so for the explanations in Sec. 6 we rely on measurements taken over thousands of training groups rather than on test accuracy alone.

4.4 Why PPO and DPO are not in the experiment

PPO. A fair PPO arm would require tuning λ and the critic’s learning rate at that scale, which our budget did not allow. On a task where every answer is two short lines, PPO’s main advantage, per-token credit, has little to work on. Critic-free and critic-based methods at scale are compared in [7], [19], [20]. We measured PPO’s memory cost instead (Sec. 6.3).

DPO. DPO learns from pairs in which one answer is marked as better (Sec. 3.3). Our task has an exact checker, so the natural signal is the check itself. A preference pair built from the checker would only carry the answer key indirectly, and a pair built by a judge would replace an exact reward with an approximate one. DPO is therefore described for context and not run. Sec. 7 says when it is the right choice.

5 Results #

With greedy decoding, the base model scores 52.3% on the 195 test questions and 77.8% on the 239 transfer questions. No model was selected on the test set.

5.1 Every variant improved the model

Table 5. Test accuracy after 80 steps (mean of three seeds, greedy decoding) and the gain over the base model.

Seven of the eight variants beat the base model, with p-values between 0.007 and 0.041. G = 16 gained the same 7.0 points with a wider interval. The other goals were learned by every variant: 99.7–100% of test answers were well formatted and short.

On the transfer set (Sec. 4.1), accuracy moved by between 1.5 and 4 points, within noise for most variants: training on the task did not cost the model its other knowledge, with one exception. One of the three runs of the variant with the larger learning rate shortened its reasons to a single word, and on the transfer set it kept the two-line format in only 45% of answers.

The brevity reward is passed by any reason of up to 23 words, so it gives no signal against going shorter, and the larger updates overshot. The other two runs of that variant, and all runs of the other variants, kept the format.

5.2 The variants were within noise of each other

Table 6. The planned comparisons between variants (difference b − a in points over matched seeds, with its 95% interval).

Every interval includes zero, so no difference between variants is established. The largest, GDPO against GRPO at equal weights, is +5.6 points with an interval of ±11.3. Removing the 20 test questions currently under revision changes no result by more than one point.

5.3 What the model learned and when

Figure 2. Share of training answers that pass each goal (mean over seeds, 95% band).

The three goals were learned in a clear order (Figure 2):

  1. The format was already right in 99% of answers at the start.
  2. Brevity was learned within about 15 steps: the average reason fell from 26 words to between 6 and 15.
  3. Correctness improved slowly and was still rising at step 80, which suggests that the training budget, rather than the choice of method, limited the final accuracy.

6 What explains the results #

The observations in this section come from the training runs themselves. We logged the rewards and advantages of every group at every step of every run. With 4 groups per step over 80 steps and three seeds (Table 3), this yielded 960 groups per variant; for G = 16, the total was 480.

6.1 Ties between goals were short-lived

With equal weights, two answers that meet different goals can receive the same total reward: a correct but long answer (correct and well formatted) and a wrong but short one (well formatted and short) both score 2 of 3. GRPO then gives them the same advantage and cannot prefer one to the other. We call these ties.

With ranked weights (1, ½, ¼), every combination of goals has its own total, so ties cannot occur. Figure 3 measures how often ties happened in training: among pairs of answers in the same group with different goal combinations, the share that received the same advantage.

Figure 3. Pairs of answers with different goal combinations but identical advantages, per training step.

Ties appeared only in the first 15 steps, while brevity was being learned: 1.7% of such pairs under GRPO with equal weights and 0.2% under GDPO with equal weights. With ranked weights there were none. Once every answer was short and well formatted, correctness was the only goal that still varied within a group, and GRPO and GDPO received essentially the same signal. With ranked weights both reached 63.1%.

On this task, then, the tie problem was real but small and temporary. The weights still matter as a statement of priorities: with ranked weights, a correct answer always outscores a wrong one, whatever else happens.

6.2 Most groups carried no correctness signal

A group is silent for correctness when all of its G answers are right or all wrong. Before training, we predicted how many groups would be silent at the start. Each training question’s pass rate over its 16 base-model samples (Sec. 4.1) gives an estimate of how often a group of G answers to that question is unanimous, and the average over the 301 training questions is 54.6% for G = 8 and 45.8% for G = 16.

Figure 4 shows what we measured on the training batches.

Figure 4. Share of groups in which every answer has the same correctness (left) or the same total reward (right) per training step, compared with the prediction from the base model (dashed: G = 8; dotted: G = 16).

About 48% of groups were silent for correctness at the start and 60–70% by the end, as more questions became easy for the model. The total reward told a different story: early in training, it still varied within almost every group, because brevity was being learned.

Monitoring only the total reward would have suggested that nearly every question was informative, when half of them taught the model only to write shorter reasons. The training library’s built-in counter of zero-spread groups is computed on the total reward and has the same blind spot.

The two variants aimed at silent groups did reduce them. The difficulty filter lowered the silent share, averaged over the run, from 54.3% under the reference variant to 42.8%, at the cost of 2,408 extra profiling answers. That is about as many as a training run. G = 16 lowered it to 40.6% but saw half as many questions per step. Neither improved test accuracy (Table 5).

Where did the gains come from? From the medium and easy questions, those the base model already solved in some of its 16 samples: their pass rate was still rising at step 80 (Sec. 5.3). The 61 training questions the base model never solved mostly stayed unsolved; in our runs, 18–29% of them produced at least one correct answer during training.

The two observations do not conflict: there was still signal to learn from on the partly solved questions when the budget ran out, while the never-solved questions produced almost no signal within 80 steps. Whether longer training would eventually reach them is a question our runs cannot answer, and on which published results differ [21], [22].

6.3 Time and memory

The training step took about 4 seconds for every variant, and peak memory was 25.5 GiB. Across 1,896 steps, the time per step followed the number of generated tokens (Pearson r = 0.65) and not the variant (Figure 5).

Figure 5. Seconds per step against generated tokens per step, for every GRPO-family run (first step excluded).

To see what PPO’s critic would cost, we measured memory after one training step in a fresh process (Table 7). A common rule estimates 16 bytes per trained parameter in mixed-precision training with Adam (2 for bf16 weights, 2 for gradients, 4 for an fp32 master copy, and 8 for Adam’s two moments) and 2 bytes per frozen parameter [23].

Table 7. Measured memory after one step (one sequence of 512 tokens), against the 16-and-2-bytes rule.

With full fine-tuning, PPO’s critic nearly doubled the memory (1.88× GRPO with a reference). With LoRA, the frozen backbone can serve as the reference, and the critic can be a second adapter on the same backbone, as in Hydra-PPO [24]; in our measurement, this added almost nothing at rest and 4.3 GiB at peak. A separate critic, a second copy of the backbone with its own adapter, would add about 16.7 GiB by the same rule.

7 Choosing a method #

Our experiment covers one branch of the choice: a task with a checker. The rest of this section relies on what the methods are designed for and on published work, not on our runs.

What kind of feedback do you have? If a program can check the answer, reinforcement learning with a verifiable reward uses that check directly. If the feedback is a judgment that one answer is better than another, whether from people or from a judge model, the problem is preference learning, and DPO is the natural choice.

A controlled comparison on the same preference data found PPO with a learned reward model slightly ahead of DPO on average, with the quality of the preference data mattering more than the algorithm [25].

Several goals? Decide first whether they can be ranked. If they can, ranked weights under GRPO encode the ranking directly (Sec. 4.2). If they cannot, GDPO was designed for that case [9].

Before a long run. Sampling about 16 answers per question from the base model is cheap and shows how many questions have mixed outcomes. On our task, this predicted the share of silent groups at the start of training to within seven points (54.6% predicted, about 48% measured, Sec. 6.2).

8 What this means for Equixly’s security agents #

Equixly’s agents act on live targets, where the reward comes from a verifier: a check that a finding really reproduces, that a flag was captured, or that the target’s state changed. The showcase used a fixed answer key, but three points from it carry over to that setting.

The verifier defines what the model learns. A verifier that misses a working attack penalizes a good action; one that accepts an empty request rewards nothing useful. Equixly already requires every reported finding to be proven exploitable, and a verifier used as a reward needs the same standard, checked with unit tests before training starts.

Expect many silent groups and measure them per goal. A security action usually succeeds or fails outright, so many groups will be unanimous. In our runs, this was 60–70% of groups by the end of training, and the total reward hid it. Logging the silent share for each goal shows early whether training still has something to learn from.

Write the priorities into the reward. In a security agent, a confirmed finding matters more than staying brief, and staying within scope is not negotiable. Ranked weights express the first kind of priority. Hard constraints such as scope and rules of engagement are better enforced as conditions that set the reward to zero when violated, since any weighted reward term can in principle be outweighed by the others.

9 Conclusion #

On a multiple-choice security task with an answer key, eight variants of GRPO and GDPO all raised the model’s accuracy by 7 to 13 points in 80 steps, and none was reliably better than another.

Two observations account for most of what we saw: the weights given to the goals decided which answers were preferred, and most groups carried no correctness signal, which the total reward hid.

These results come from one task, one training budget, and, apart from one exploratory variant, one learning rate, and describe this showcase only. The next step is to repeat the comparison with a live verifier, longer training, and a tuned learning rate per method.

References #

[1] R. Rafailov et al., “Direct preference optimization: Your language model is secretly a reward model,” in Proc. NeurIPS, 2023, arXiv:2305.18290.

[2] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg and D. Amodei, “Deep reinforcement learning from human preferences,” in Proc. NeurIPS, 2017, arXiv:1706.03741.

[3] D. M. Ziegler et al., “Fine-tuning language models from human preferences,” [arXiv:1909.08593](https://arxiv.org/abs/1909.08593), 2019.

[4] N. Stiennon et al., “Learning to summarize from human feedback,” in *Proc. NeurIPS*, 2020, [arXiv:2009.01325](https://arxiv.org/abs/2009.01325).

[5] L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Proc. NeurIPS, 2022, arXiv:2203.02155.

[6] J. Schulman, F. Wolski, P. Dhariwal, A. Radford and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017.

[7] Z. Shao et al., “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,” [arXiv:2402.03300](https://arxiv.org/abs/2402.03300), 2024.

[8] DeepSeek-AI, “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” *Nature*, vol. 645, 2025, [arXiv:2501.12948](https://arxiv.org/abs/2501.12948).

[9] Liu et al., “GDPO: Group reward-decoupled normalization policy optimization,” in *Proc. ICML*, 2026, [arXiv:2601.05242](https://arxiv.org/abs/2601.05242).

[10] J. Schulman, P. Moritz, S. Levine, M. Jordan and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” in Proc. ICLR, 2016, arXiv:1506.02438.

[11] W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang and T. Zhang, “Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint,” arXiv:2312.11456, 2023.

[12] S. Guo et al., “Direct language model alignment from online AI feedback,” arXiv:2402.04792, 2024. [13] A. Ahmadian et al., “Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs,” in Proc. ACL, 2024, arXiv:2402.14740.

[14] Q. Yu et al., “DAPO: An open-source LLM reinforcement learning system at scale,” [arXiv:2503.14476](https://arxiv.org/abs/2503.14476), 2025.

[15] Z. Liu et al., “Understanding R1-Zero-like training: A critical perspective,” [arXiv:2503.20783](https://arxiv.org/abs/2503.20783), 2025.

[16] Hugging Face, “TRL: Transformer reinforcement learning,” version 1.15.0. [Online]. Available: [https://github.com/huggingface/trl](https://github.com/huggingface/trl) (Accessed: Oct. 27, 2026).

[17] W. Kwon et al., “Efficient memory management for large language model serving with PagedAttention,” in Proc. SOSP, 2023, arXiv:2309.06180.

[18] S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979.

[19] A. Kazemnejad et al., “VinePPO: Unlocking RL potential for LLM reasoning through refined credit assignment,” [arXiv:2410.01679](https://arxiv.org/abs/2410.01679), 2024.

[20] Y. Yue et al., “VAPO: Efficient and reliable reinforcement learning for advanced reasoning tasks,” [arXiv:2504.05118](https://arxiv.org/abs/2504.05118), 2025.

[21] Y. Yue et al., “Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?,” arXiv:2504.13837, 2025.

[22] M. Liu et al., “ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language models,” arXiv:2505.24864, 2025. [23] S. Rajbhandari, J. Rasley, O. Ruwase and Y. He, “ZeRO: Memory optimizations toward training trillion parameter models,” in Proc. SC, 2020, arXiv:1910.02054.

[24] M. Santacroce, Y. Lu, H. Yu, Y. Li and Y. Shen, “Efficient RLHF: Reducing the memory usage of PPO,” arXiv:2309.00754, 2023. [25] H. Ivison et al., “Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback,” in Proc. NeurIPS, 2024, arXiv:2406.09279.

── more in #ai-research 4 stories · sorted by recency
── more on @equixly 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/when-the-answer-can-…] indexed:0 read:28min 2026-11-17 · —