Embarrassingly Simple Self-Distillation Improves Code Generation

Apple · 2 months ago · 55

Researchers found that large language models can improve their code generation performance through simple self-distillation, which involves sampling the model's own outputs with specific temperature settings and then fine-tuning on those samples. The method improved Qwen3-30B-Instruct's pass@1 score on LiveCodeBench from 42.4% to 55.3%, with greater improvements on harder problems. The approach generalizes across different model families and sizes without requiring external verifiers, teacher models, or reinforcement learning, offering a new post-training direction for enhancing LLM code generation.