We talked about this sort of thing a bit before, but now the official Multi-Teacher On-Policy distillation paper is out, and its a pleasant read: “MOPD for Capability Integration in LLM Post-Training”.
The problem MOPD is solving is composing a bunch of different capabilities into the same model. Normally you do this with RL, with different pipelines for different kinds of capabilities:
Each pipeline reliably improves the model’s capability on its target domain. However, what we ultimately want is a single model that performs well across all of these domains. Yet building such a model remains an open problem in modern LLM post-training.
Because RL training is fairly superficial, it’s easy for the capabilities to get (partially) overridden by later ones. To avoid this kind of interference, MOPD runs the RL pipelines independently. You create multiple domain-expert teachers from the same base model, then use self-distillation across a mix of tasks scoring each entry in the batch with the appropriate teacher, to train a composite student model.
To get a feel for this, I tried it on my FactWorld eval and Qwen3-1.7B with a LoRA adapter. I RL’d two tasks into two different LoRA adapters, binding 1 and recall
, and then created a fresh adapter identical to baseline to be the student.
2The MOPD process works by grabbing a batch that includes both binding and recall questions at once.The student answers each one, generating its own predicted tokens. This is what makes it “on-policy”: the training data is the outputs from the model itself. The answers in the batch are routed to their appropriate teachers 3. The teacher calculates probabilities for all of the tokens the student generated, including its full opinion over the vocab for each position.
We then measure the gap. The paper offers two different ways to do this: PG 4, which adjusts the probabilities kust for the tokens the student actually chose, and KL
, which does the entire token distribution. Both the binding and recall gaps are backpropped into the single student adapter in one step. Rinse, repeat.
5The end product is one student adapter that behaves like the binding teacher on binding questions and like the recall teacher on recall questions. And, as multiple models have shown now, you can scale this up a very long way.
- Last-write-wins state tracking. You see a stream of a gives x to b type strings, and it asks “who is the holder of x”
[↩︎](#31483019-d470-4ee6-87a1-7c9f05ca097e-link) - You get a long list of facts like “a’s x is y” with different owners, and are asked “what is x of a”
[↩︎](#79ba5c91-ae83-4c94-9318-9d30421e6fb5-link) - In my implementation this is actually just swapping LoRAs because lazy
[↩︎](#a2cad023-45cb-43f9-afb0-96d988a5db69-link) - Policy Gradient, an approach from the PPO RL method
[↩︎](#c2f3cebb-8846-4698-9e1f-54e40d3428ae-link) - Reverse Kullback-Leibler divergence, in my case full, in the papers case top-k entries
[↩︎](#7f4778d1-8cbc-4907-aaa9-c375763cde0d-link)