This digest covers post-training research and engineering posts published between 2026-10-09 and 2026-10-16: new arXiv papers on distillation, reward modeling, and RL post-training, plus blog posts from Hugging Face and a roundup from Latent Space's daily AI-news digest.
Imprinting: Knowledge Distillation Can Shift Student Behavior — distilling on unrelated tasks still leaks the teacher's safety disposition.
Apertus 1.5 Release Notes — a fully documented SFT → RLVR → DPO → SDPO post-training stack. RM-OPD: On-Policy Distillation for Generative Reward Models — a distilled reward model that beats the specialist teachers it learned from.
UltraBERT: One Encoder for Classification and Retrieval — cascade distillation helps classification but hurts retrieval at high masking.
[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment — a cheap LoRA+GRPO recipe, a multi-teacher distillation pipeline, and a new LoRA optimizer, all in one digest.
Imprinting: Knowledge Distillation Can Shift Student Behavior — 2026-10-13 Distilling a student from a teacher's outputs on tasks completely unrelated to safety still transfers the teacher's safety disposition: a harm-accepting teacher measurably reduces the student's harm avoidance, and vice versa. Token-removal weakens the effect but doesn't eliminate it, and a post-hoc DPO pass only partially corrects it. Practical takeaway: audit a distillation teacher's behavior, not just its capability, before using it — downstream alignment steps won't fully undo what got imprinted.
RM-OPD: On-Policy Distillation for Generative Reward Models — 2026-10-14 (updated 2026-10-15) Trains domain-specialized reward model teachers first, then consolidates them into one generative student via routed on-policy distillation with dense token-level supervision on the student's own critiques. The unified student reportedly surpasses the specialized teachers it was distilled from. Useful for anyone merging reward models across heterogeneous domains for RLHF/RLAIF without the usual accuracy hit from naive consolidation.
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF — 2026-10-15 Uses a small clean preference subset to calibrate per-sample noise via optimal transport, aligning the noisy label distribution to the clean one, then trains the reward model with a noise-corrected loss that's unbiased relative to clean data. Directly actionable for teams running RLHF on cheap-but-noisy crowd or weak-LLM-judge labels — it offers a path to reward-model quality without a fully clean, expensive labeling pass.
VASPO: Self-Distillation with Privileged Value Estimation — 2026-10-14 Combines self-teacher distillation signals with a critic that uses end-of-response (privileged) information to sharpen token-level advantage estimates inside a PPO loop, reporting +2.59 and +1.81 points over standard PPO on Qwen3-4B and OLMo3-7B math reasoning benchmarks. A drop-in PPO improvement (code released) for teams already running RL post-training who aren't ready to switch to GRPO or DPO.
Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints — 2026-10-13 Introduces SeTPO: reverse-KL teacher feedback injected into the policy's advantages only on low-reward trajectories, plus a sustained forward-KL term to retain demonstrated behavior, fixing a known failure mode where naive teacher supervision during RL misaligns small models on tool use. Outperforms plain SFT, RL, and vanilla distillation on BFCL v4 and API-Bank — relevant if RL-only training is producing brittle tool-calling policies.
Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry — 2026-10-14 Both preference optimization and trajectory RL fail to credit the local transition that turns a bad intermediate reasoning state into a better one; LSPO adds a learned "lifted" coordinate and energy function to internalize that correction directly into the generator. On Qwen3.5-27B it matches test-time reflection accuracy (71.8% vs 71.9%) with 28% lower latency and 25.8% fewer tokens in a single forward pass — a real trade-off against relying on inference-time reflection scaffolding.
Apertus 1.5 Release Notes — 2026-10-15 Documents Apertus 1.5's full post-training pipeline end to end: SFT on 3.7M text/image/speech samples, RLVR (RL with verifiable rewards) on 350k math/code/instruction prompts, DPO on LLM-judged preference pairs, and a custom SDPO stage that aligns the model to the "Apertus Charter" via judge feedback plus self-distillation, with synthetic LongContextQA data for context extension. A rare fully-documented four-stage alignment stack with real dataset sizes — a reference point for deciding how many alignment stages are worth stacking and in what order.
UltraBERT: One Encoder for Classification and Retrieval — 2026-10-14 Uses cascade distillation (a 1.3B teacher distilled in stages into 380M and 150M students) and finds that distillation helps classification but hurts retrieval at high masking rates — a counterintuitive result that should change training strategy depending on the target task. Also reports UltraBERT needs learning rates up to 10x smaller than other encoders, plus a concrete GLUE gain (90.2 → 90.8 on the 380M model) from the cascade approach.
Nothing new on post-training this week — the one post published in the window ("I expect rapid progress but not towards general superintelligence," 2026-10-09) is about AI timelines, not fine-tuning technique.
Nothing relevant this week — the only posts in the window were hosting announcements for Meta's Muse Spark Open model, with no post-training technique discussion.
[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment — 2026-10-16 This daily digest bundles three concrete post-training data points: a LoRA+GRPO run (verl for training, vLLM for rollouts) that lifted AIME-2024 mean@8 from 0.56 to 0.85 over 150 steps on 16 GB300s; Nemotron 3.5's recipe of 1T-token distillation from a larger Nemotron-3 teacher plus multi-teacher MOPD post-training and an MTP-boosting step that raised acceptance length 21.8% to 4.61; and Thinking Machines' "Dimuon," a Muon-variant optimizer for LoRA fine-tuning shipped in Tinker. Three actionable reference points in one place — a cheap GRPO+LoRA recipe with a before/after benchmark, a multi-teacher distillation pipeline, and a new LoRA-friendly optimizer worth benchmarking against AdamW.
Distillation is this week's recurring character, but from opposite angles: the arXiv papers warn that it silently imprints behavior you didn't ask for (Imprinting) and that merging it carelessly across domains loses accuracy (RM-OPD's fix), while the engineering posts (Apertus, UltraBERT, Nemotron 3.5) show it's still the cheapest way to compress scale into a smaller model — provided you watch what, specifically, transfers. Pair that with CleanRM and RM-OPD both targeting reward-model quality from imperfect inputs, and the throughline is clear: teams are spending more effort on the supervision signal itself — who the teacher is, how clean the preferences are — than on the RL algorithm consuming it.
What's catching your eye in post-training this week? Drop a comment below.