Embarrassingly Simple Self-Distillation Improves Code Generation

Apple's ML research team just published something that sounds almost too simple to work: an AI model getting better at writing code by studying its own homework. No teacher model, no reward signal, no reinforcement learning loop. Just sample a bunch of outputs, pick some, fine-tune on them, repeat.

The technique is called SSD, short for simple self-distillation, and the results are hard to wave away. Qwen3-30B-Instruct jumped from 42.4% to 55.3% pass@1 on LiveCodeBench v6, a serious benchmark for code generation, purely from chewing on its own generated samples with specific temperature and truncation settings. The team, led by Ruixiang Zhang and five co-authors, saw the biggest gains on the hardest problems in the set, which is exactly where you'd expect a model to need outside help, not a mirror.

And it wasn't a one-model fluke. The researchers ran SSD across Qwen and Llama families at 4B, 8B, and 30B parameter scales, hitting both instruct-tuned and reasoning-focused 'thinking' variants. Gains showed up everywhere. That kind of consistency across architectures and sizes is what turns a curious result into something worth building on.

The explanation the team offers is where this gets genuinely interesting. They point to what they call a precision-exploration conflict baked into how LLMs decode text: sometimes the model needs to commit hard to one likely answer, other times it needs to keep several options alive to explore. SSD, they argue, learns to tell these situations apart. It trims away noisy, low-value alternatives when precision counts, but leaves the useful spread of options intact when exploration actually helps. In effect, the model is sharpening its own judgment about when to be decisive and when to stay flexible, using nothing but its own sampled behavior as the training signal.

Apple frames SSD as a complementary direction for post-training, not a replacement for RLHF or verifier-based approaches. Given how expensive those methods are to run at scale, a technique that squeezes real gains out of a model's own outputs, with standard supervised fine-tuning and no extra infrastructure, is the kind of low-cost lever labs will want to pull first.