Embarrassingly Simple Self-Distillation Improves Code Generation
Apple ML Research
Researchers found that large language models can improve their code generation performance through simple self-distillation, which involves sampling the model's own outputs with specific temperature settings and then fine-tuning on those samples. The method improved Qwen3-30B-Instruct's pass@1 score on LiveCodeBench from 42.4% to 55.3%, with greater improvements on harder problems. The approach generalizes across different model families and sizes without requiring external verifiers, teacher models, or reinforcement learning, offering a new post-training direction for enhancing LLM code generation.
Why it matters
Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both…
