{"slug": "do-your-capabilities-homework", "title": "Do your capabilities homework", "summary": "A technical AI safety researcher argues that safety-focused researchers should engage with capabilities research, highlighting On-Policy Self-Distillation (OPSD) as a promising alternative to GRPO for training reasoning models. The post, referencing DeepSeek's R1 paper from 1.5 years ago, explains how OPSD uses a teacher-student setup to provide token-level feedback, potentially avoiding the blunt all-or-nothing updates of GRPO that can harm model capabilities. The author contends that such methods could lead to safer AI as a consequence of the learning algorithm itself.", "body_md": "It seems to me that *a lot* of technical ai safety people haven't done their capabilities homework - and that's a shame! I'll try to illuminate here mainly with an example as to why I think people who care about safety should totally pay more attention to the trends and actively engage with them - the case for safe AI not through an additional loss term but as a consequence of the learning algorithm!\n\nIt's now been 1.5 years since [R1 ](https://arxiv.org/abs/2501.12948)came out - the paper which really introduced RLVR (RL with verifiable rewards) through GRPO at scale. GRPO is stupidly simple, reminding of early REINFORCE algorithms: sample n traces, assign them a reward and make the advantage a normalized version of their reward, applied to the whole trace. In other words: for a trace which resulted in a correct final answer, slightly increase the probability of sampling each token of its trace and vice versa.\n\nThis is also what safety focused people generally engage with - and that's totally fair! While GRPO has gone through some variations since then ([Dr. GRPO](https://arxiv.org/abs/2503.20783), [DAPO](https://arxiv.org/abs/2503.14476), ...), these are mostly minor improvements that you should not waste your time on.\n\nI further agree that this form of training is incredibly dangerous - we seem to now be reaching the amount of post-training required to meaningfully differ from the benign prior, and it's [not exactly looking peachy](https://openai.com/de-DE/index/hugging-face-model-evaluation-security-incident/).\n\n**Yet**, it should be clear that GRPO also seems comically terrible from a capabilities standpoint. So has the capabilities-front been cooking up anything that might interest safety researchers as well? The answer is yes!\n\nI will keep this at a high-level:\n\nLet's walk through this with an example:\n\nWe have some math problem, say \"What's 32 * 5?\". We generate a completion, which turns out to be wrong:\n\nLet's think about this step by step. [...] 30 * 5 + 2\n\n*5 = 140 + 10 = 150. Therefore the final answer is 150.\n\nNotice that 30 * *5 *is not 140, but everything else was perfectly reasonable arithmetic. Standard GRPO would now lower the probability of sampling all of these tokens even though there is a single error - everything else is fine and shouldn't be disturbed!\n\nNow let's walk through On-Policy Self-Distillation (OPSD):\n\nWe can reuse the same student completion as example; after all, we are now also On-Policy. But this time we don't have some kind of binary reward for the entire trace - instead we create a second prompt for our 'fake' teacher model:\n\nSolve the following problem: What's 32 *\n\n5?\n\nIf you get stuck, you may use the following example solution as help, but first try it yourself:\n\n32 * 5 = (30 + 2) * 5 = 30 *5 + 2 * 5 = 150 + 10 = 160. Final answer: 160\n\nNow, we aren't doing normal off-policy distillation, so we won't be using this prompt to actually generate anything. Instead we concatenate the student's attempt below!\n\nSolve the following problem: What's 32 * 5?\n\nIf you get stuck, you may use the following example solution as help, but first try it yourself:\n\n32*5 = (30 + 2) * 5 = 30 *5 + 2 * 5 = 150 + 10 = 160. Final answer: 160Let's think about this step by step. [...] 30 * 5 + 2 *\n\n5 = 140 + 10 = 150. Therefore the final answer is 150.\n\nFinally we do distillation over these generated student tokens: the context window with extra information acting as teacher, and the one without acting as student. The teacher will recognize that up to \"30 * 5 + 2 ** *5 =\" everything is reasonable and not meaningfully disturb the student. Right before the '140' token, their predictions will massively diverge: even if the teacher is the same model, which might very well make this mistake, it now knows for a fact that it isn't correct. Down the line, say right before 'Therefore', it might also start trying to reverse the error, predicting tokens like 'Wait'/'But' in an attempt to correct the last step.\n\nI hope the elegance of this approach becomes clear. It should remind you of how humans think about their past mistakes: with your new knowledge of what actually turned out to be a more worthwhile solution, you revisit your previous steps and consider where you went down the wrong path and should have explored other directions.\n\nBetween RLVR and OPSD - which would you prefer to be used to train frontier models? Assume that in raw intelligence, they would be the same.\n\nI hope by now you shouted 'On-Policy Self-Distillation', even if only in your head:\n\nDo I think this approach is perfect? No.\n\nDo I think it's 100x safer than normal RLVR? Yes, GOD YES.\n\nSo where does this approach currently stand - can we imagine it taking over RLVR soon?\n\nI described in [early December of last year the vision](https://www.lesswrong.com/posts/HCo4FhiahuNiq39Cb/grpo-is-terrible) for precisely this algorithm (and more). In late January, [a paper first appeared on arxiv](https://arxiv.org/abs/2601.18734) which seems to have independently discovered & implemented it and the results were promising. Since then, there have been quite a lot of papers on it, many different variations (of which, sadly, a lot are slop). Nevertheless, we seem to be making progress.\n\nIn my post from December I already laid out the problem of 'Impossible Knowledge'. By this I refer to the fact that our teacher could just directly output \"160\", simply because it already knows stuff it shouldn't know yet. Teaching the student to imitate this 'Impossible Knowledge' will be pointless and not help capabilities once this knowledge is gone. I proposed an initial solution, and it seems to have been tried out[ just 2 weeks ago](https://arxiv.org/abs/2607.02502): the empirical results are too short for me to make clear statements, but initial results do seem promising.\n\nIn terms of frontier labs adopting it: it seems Cursor is actively using it for training its in-house models with the caveat that they are using *both* RLVR and OPSD - which mostly ruins the point from a safety perspective.\n\nThere is much slop and many incremental results in research of LLM capabilities - I do not think technical AI Safety researchers should waste their time on such results. But, from time to time, there are meaningful new directions and AI Safety researchers can NOT be the last to find out.\n\nJust a few days ago, it seems to have come as a shock that Kimi K3 uses a residual stream which has an additional routing mechanism, complicating many interpretability approaches. Some people even initially believed that this was a good thing for the mech interp faction!\n\nThis kind of routing is NOT new and has been [proposed ](https://arxiv.org/abs/2512.24880)[and ](https://arxiv.org/abs/2603.15619)[tested ](https://arxiv.org/abs/2603.15031)before, even at meaningful scale. Kimi K3 was simply the first real frontier-model (in terms of open-weights) to adopt it. I would have wished that especially mech interp researchers were more aware of such approaches because it's not unlikely that at least one frontier lab is by now using something alike internally.\n\nBut even aside from the more depressing side of labs chasing capability at almost any cost and safety researchers having to always be a step ahead as a result - being on top of capability research can also provide more optimistic sides, as I hope the example in this post could show.", "url": "https://wpnews.pro/news/do-your-capabilities-homework", "canonical_source": "https://www.lesswrong.com/posts/dYnhhTxoDj3fuCxLB/do-your-capabilities-homework", "published_at": "2026-08-01 16:10:57+00:00", "updated_at": "2026-08-01 16:30:56.197388+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-safety"], "entities": ["DeepSeek", "GRPO", "On-Policy Self-Distillation", "R1", "Dr. GRPO", "DAPO"], "alternates": {"html": "https://wpnews.pro/news/do-your-capabilities-homework", "markdown": "https://wpnews.pro/news/do-your-capabilities-homework.md", "text": "https://wpnews.pro/news/do-your-capabilities-homework.txt", "jsonld": "https://wpnews.pro/news/do-your-capabilities-homework.jsonld"}}