# Embarrassingly Simple Self-Distillation Improves Code Generation

> Source: <https://machinelearning.apple.com/research/simple-self-distillation>
> Published: 2026-07-16 00:00:00+00:00

[content type paper](/research/)published July 2026

Embarrassingly Simple Self-Distillation Improves Code Generation

AuthorsRuixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang

Embarrassingly Simple Self-Distillation Improves Code Generation

AuthorsRuixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang

Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation.

BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks

August 3, 2024[research area Human-Computer Interaction](/research/?domain=Human-Computer%20Interaction), [research area Tools, Platforms, Frameworks](/research/?domain=Tools%2C%20Platforms%2C%20Frameworks)[conference IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)](/research/?event=IEEE%20Symposium%20on%20Visual%20Languages%20and%20Human-Centric%20Computing%20(VL%2FHCC))

This paper was accepted at IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) 2024.

Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on large language models (LLMs). However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into…

Applying RLAIF for Code Generation with API-usage in Lightweight LLMs

July 1, 2024[research area Methods and Algorithms](/research/?domain=Methods%20and%20Algorithms), [research area Speech and Natural Language Processing](/research/?domain=Speech%20and%20Natural%20Language%20Processing)[Workshop at ACL](/research/?event=ACL%20Workshop)

This paper was accepted at the Natural Language Reasoning and Structured Explanations workshop at ACL 2024.

Reinforcement Learning from AI Feedback (RLAIF) has demonstrated significant potential across various domains, including mitigating harm in LLM outputs, enhancing text summarization, and mathematical reasoning. This paper introduces an RLAIF framework for improving the code generation abilities of lightweight (<1B parameters) LLMs. We…