Do your capabilities homework

A technical AI safety researcher argues that safety-focused researchers should engage with capabilities research, highlighting On-Policy Self-Distillation (OPSD) as a promising alternative to GRPO for training reasoning models. The post, referencing DeepSeek's R1 paper from 1.5 years ago, explains how OPSD uses a teacher-student setup to provide token-level feedback, potentially avoiding the blunt all-or-nothing updates of GRPO that can harm model capabilities. The author contends that such methods could lead to safer AI as a consequence of the learning algorithm itself.

It seems to me that a lot of technical ai safety people haven't done their capabilities homework - and that's a shame I'll try to illuminate here mainly with an example as to why I think people who care about safety should totally pay more attention to the trends and actively engage with them - the case for safe AI not through an additional loss term but as a consequence of the learning algorithm It's now been 1.5 years since R1 https://arxiv.org/abs/2501.12948 came out - the paper which really introduced RLVR RL with verifiable rewards through GRPO at scale. GRPO is stupidly simple, reminding of early REINFORCE algorithms: sample n traces, assign them a reward and make the advantage a normalized version of their reward, applied to the whole trace. In other words: for a trace which resulted in a correct final answer, slightly increase the probability of sampling each token of its trace and vice versa. This is also what safety focused people generally engage with - and that's totally fair While GRPO has gone through some variations since then Dr. GRPO https://arxiv.org/abs/2503.20783 , DAPO https://arxiv.org/abs/2503.14476 , ... , these are mostly minor improvements that you should not waste your time on. I further agree that this form of training is incredibly dangerous - we seem to now be reaching the amount of post-training required to meaningfully differ from the benign prior, and it's not exactly looking peachy https://openai.com/de-DE/index/hugging-face-model-evaluation-security-incident/ . Yet , it should be clear that GRPO also seems comically terrible from a capabilities standpoint. So has the capabilities-front been cooking up anything that might interest safety researchers as well? The answer is yes I will keep this at a high-level: Let's walk through this with an example: We have some math problem, say "What's 32 5?". We generate a completion, which turns out to be wrong: Let's think about this step by step. ... 30 5 + 2 5 = 140 + 10 = 150. Therefore the final answer is 150. Notice that 30 5 is not 140, but everything else was perfectly reasonable arithmetic. Standard GRPO would now lower the probability of sampling all of these tokens even though there is a single error - everything else is fine and shouldn't be disturbed Now let's walk through On-Policy Self-Distillation OPSD : We can reuse the same student completion as example; after all, we are now also On-Policy. But this time we don't have some kind of binary reward for the entire trace - instead we create a second prompt for our 'fake' teacher model: Solve the following problem: What's 32 5? If you get stuck, you may use the following example solution as help, but first try it yourself: 32 5 = 30 + 2 5 = 30 5 + 2 5 = 150 + 10 = 160. Final answer: 160 Now, we aren't doing normal off-policy distillation, so we won't be using this prompt to actually generate anything. Instead we concatenate the student's attempt below Solve the following problem: What's 32 5? If you get stuck, you may use the following example solution as help, but first try it yourself: 32 5 = 30 + 2 5 = 30 5 + 2 5 = 150 + 10 = 160. Final answer: 160Let's think about this step by step. ... 30 5 + 2 5 = 140 + 10 = 150. Therefore the final answer is 150. Finally we do distillation over these generated student tokens: the context window with extra information acting as teacher, and the one without acting as student. The teacher will recognize that up to "30 5 + 2 5 =" everything is reasonable and not meaningfully disturb the student. Right before the '140' token, their predictions will massively diverge: even if the teacher is the same model, which might very well make this mistake, it now knows for a fact that it isn't correct. Down the line, say right before 'Therefore', it might also start trying to reverse the error, predicting tokens like 'Wait'/'But' in an attempt to correct the last step. I hope the elegance of this approach becomes clear. It should remind you of how humans think about their past mistakes: with your new knowledge of what actually turned out to be a more worthwhile solution, you revisit your previous steps and consider where you went down the wrong path and should have explored other directions. Between RLVR and OPSD - which would you prefer to be used to train frontier models? Assume that in raw intelligence, they would be the same. I hope by now you shouted 'On-Policy Self-Distillation', even if only in your head: Do I think this approach is perfect? No. Do I think it's 100x safer than normal RLVR? Yes, GOD YES. So where does this approach currently stand - can we imagine it taking over RLVR soon? I described in early December of last year the vision https://www.lesswrong.com/posts/HCo4FhiahuNiq39Cb/grpo-is-terrible for precisely this algorithm and more . In late January, a paper first appeared on arxiv https://arxiv.org/abs/2601.18734 which seems to have independently discovered & implemented it and the results were promising. Since then, there have been quite a lot of papers on it, many different variations of which, sadly, a lot are slop . Nevertheless, we seem to be making progress. In my post from December I already laid out the problem of 'Impossible Knowledge'. By this I refer to the fact that our teacher could just directly output "160", simply because it already knows stuff it shouldn't know yet. Teaching the student to imitate this 'Impossible Knowledge' will be pointless and not help capabilities once this knowledge is gone. I proposed an initial solution, and it seems to have been tried out just 2 weeks ago https://arxiv.org/abs/2607.02502 : the empirical results are too short for me to make clear statements, but initial results do seem promising. In terms of frontier labs adopting it: it seems Cursor is actively using it for training its in-house models with the caveat that they are using both RLVR and OPSD - which mostly ruins the point from a safety perspective. There is much slop and many incremental results in research of LLM capabilities - I do not think technical AI Safety researchers should waste their time on such results. But, from time to time, there are meaningful new directions and AI Safety researchers can NOT be the last to find out. Just a few days ago, it seems to have come as a shock that Kimi K3 uses a residual stream which has an additional routing mechanism, complicating many interpretability approaches. Some people even initially believed that this was a good thing for the mech interp faction This kind of routing is NOT new and has been proposed https://arxiv.org/abs/2512.24880 and https://arxiv.org/abs/2603.15619 tested https://arxiv.org/abs/2603.15031 before, even at meaningful scale. Kimi K3 was simply the first real frontier-model in terms of open-weights to adopt it. I would have wished that especially mech interp researchers were more aware of such approaches because it's not unlikely that at least one frontier lab is by now using something alike internally. But even aside from the more depressing side of labs chasing capability at almost any cost and safety researchers having to always be a step ahead as a result - being on top of capability research can also provide more optimistic sides, as I hope the example in this post could show.