Distillation's Hidden Costs and Payoffs — Teachers Leak Behavior, Reward Models Surpass Them

A roundup of post-training research published between October 9 and 16, 2026 reports that knowledge distillation can imprint a teacher model's safety disposition onto students even when training on unrelated tasks, and that a routed on-policy distillation method (RM-OPD) produces a unified generative reward model that surpasses the specialized teachers it was distilled from. Other papers cover noise-corrected reward modeling from a small clean preference subset (CleanRM), self-distillation with privileged value estimation for PPO (VASPO, reporting +2.59 and +1.81 points on Qwen3-4B and OLMo3-7B math reasoning), and bidirectional KL constraints for teacher-guided tool learning in small models.

This digest covers post-training research and engineering posts published between 2026-10-09 and 2026-10-16: new arXiv papers on distillation, reward modeling, and RL post-training, plus blog posts from Hugging Face and a roundup from Latent Space's daily AI-news digest. Imprinting: Knowledge Distillation Can Shift Student Behavior https://arxiv.org/abs/2610.18517 — distilling on unrelated tasks still leaks the teacher's safety disposition. Apertus 1.5 Release Notes https://huggingface.co/blog/swiss-ai/apertus-v15-release-notes — a fully documented SFT → RLVR → DPO → SDPO post-training stack. RM-OPD: On-Policy Distillation for Generative Reward Models https://arxiv.org/abs/2610.20174 — a distilled reward model that beats the specialist teachers it learned from. UltraBERT: One Encoder for Classification and Retrieval https://huggingface.co/blog/dmis-lab/ultrabert — cascade distillation helps classification but hurts retrieval at high masking. \ AINews\ Agents disprove Polysemanticity; greatly improve Interpretability & Alignment https://www.latent.space/p/ainews-agents-disprove-polysemanticity — a cheap LoRA+GRPO recipe, a multi-teacher distillation pipeline, and a new LoRA optimizer, all in one digest. Imprinting: Knowledge Distillation Can Shift Student Behavior https://arxiv.org/abs/2610.18517 — 2026-10-13 Distilling a student from a teacher's outputs on tasks completely unrelated to safety still transfers the teacher's safety disposition: a harm-accepting teacher measurably reduces the student's harm avoidance, and vice versa. Token-removal weakens the effect but doesn't eliminate it, and a post-hoc DPO pass only partially corrects it. Practical takeaway: audit a distillation teacher's behavior , not just its capability, before using it — downstream alignment steps won't fully undo what got imprinted. RM-OPD: On-Policy Distillation for Generative Reward Models https://arxiv.org/abs/2610.20174 — 2026-10-14 updated 2026-10-15 Trains domain-specialized reward model teachers first, then consolidates them into one generative student via routed on-policy distillation with dense token-level supervision on the student's own critiques. The unified student reportedly surpasses the specialized teachers it was distilled from. Useful for anyone merging reward models across heterogeneous domains for RLHF/RLAIF without the usual accuracy hit from naive consolidation. CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF https://arxiv.org/abs/2610.21138 — 2026-10-15 Uses a small clean preference subset to calibrate per-sample noise via optimal transport, aligning the noisy label distribution to the clean one, then trains the reward model with a noise-corrected loss that's unbiased relative to clean data. Directly actionable for teams running RLHF on cheap-but-noisy crowd or weak-LLM-judge labels — it offers a path to reward-model quality without a fully clean, expensive labeling pass. VASPO: Self-Distillation with Privileged Value Estimation https://arxiv.org/abs/2610.20516 — 2026-10-14 Combines self-teacher distillation signals with a critic that uses end-of-response privileged information to sharpen token-level advantage estimates inside a PPO loop, reporting +2.59 and +1.81 points over standard PPO on Qwen3-4B and OLMo3-7B math reasoning benchmarks. A drop-in PPO improvement code released for teams already running RL post-training who aren't ready to switch to GRPO or DPO. Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints https://arxiv.org/abs/2610.18459 — 2026-10-13 Introduces SeTPO: reverse-KL teacher feedback injected into the policy's advantages only on low-reward trajectories, plus a sustained forward-KL term to retain demonstrated behavior, fixing a known failure mode where naive teacher supervision during RL misaligns small models on tool use. Outperforms plain SFT, RL, and vanilla distillation on BFCL v4 and API-Bank — relevant if RL-only training is producing brittle tool-calling policies. Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry https://arxiv.org/abs/2610.20350 — 2026-10-14 Both preference optimization and trajectory RL fail to credit the local transition that turns a bad intermediate reasoning state into a better one; LSPO adds a learned "lifted" coordinate and energy function to internalize that correction directly into the generator. On Qwen3.5-27B it matches test-time reflection accuracy 71.8% vs 71.9% with 28% lower latency and 25.8% fewer tokens in a single forward pass — a real trade-off against relying on inference-time reflection scaffolding. Apertus 1.5 Release Notes https://huggingface.co/blog/swiss-ai/apertus-v15-release-notes — 2026-10-15 Documents Apertus 1.5's full post-training pipeline end to end: SFT on 3.7M text/image/speech samples, RLVR RL with verifiable rewards on 350k math/code/instruction prompts, DPO on LLM-judged preference pairs, and a custom SDPO stage that aligns the model to the "Apertus Charter" via judge feedback plus self-distillation, with synthetic LongContextQA data for context extension. A rare fully-documented four-stage alignment stack with real dataset sizes — a reference point for deciding how many alignment stages are worth stacking and in what order. UltraBERT: One Encoder for Classification and Retrieval https://huggingface.co/blog/dmis-lab/ultrabert — 2026-10-14 Uses cascade distillation a 1.3B teacher distilled in stages into 380M and 150M students and finds that distillation helps classification but hurts retrieval at high masking rates — a counterintuitive result that should change training strategy depending on the target task. Also reports UltraBERT needs learning rates up to 10x smaller than other encoders, plus a concrete GLUE gain 90.2 → 90.8 on the 380M model from the cascade approach. Nothing new on post-training this week — the one post published in the window "I expect rapid progress but not towards general superintelligence," 2026-10-09 is about AI timelines, not fine-tuning technique. Nothing relevant this week — the only posts in the window were hosting announcements for Meta's Muse Spark Open model, with no post-training technique discussion. \ AINews\ Agents disprove Polysemanticity; greatly improve Interpretability & Alignment https://www.latent.space/p/ainews-agents-disprove-polysemanticity — 2026-10-16 This daily digest bundles three concrete post-training data points: a LoRA+GRPO run verl for training, vLLM for rollouts that lifted AIME-2024 mean@8 from 0.56 to 0.85 over 150 steps on 16 GB300s; Nemotron 3.5's recipe of 1T-token distillation from a larger Nemotron-3 teacher plus multi-teacher MOPD post-training and an MTP-boosting step that raised acceptance length 21.8% to 4.61; and Thinking Machines' "Dimuon," a Muon-variant optimizer for LoRA fine-tuning shipped in Tinker. Three actionable reference points in one place — a cheap GRPO+LoRA recipe with a before/after benchmark, a multi-teacher distillation pipeline, and a new LoRA-friendly optimizer worth benchmarking against AdamW. Distillation is this week's recurring character, but from opposite angles: the arXiv papers warn that it silently imprints behavior you didn't ask for Imprinting and that merging it carelessly across domains loses accuracy RM-OPD's fix , while the engineering posts Apertus, UltraBERT, Nemotron 3.5 show it's still the cheapest way to compress scale into a smaller model — provided you watch what, specifically, transfers. Pair that with CleanRM and RM-OPD both targeting reward-model quality from imperfect inputs, and the throughline is clear: teams are spending more effort on the supervision signal itself — who the teacher is, how clean the preferences are — than on the RL algorithm consuming it. What's catching your eye in post-training this week? Drop a comment below.