Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova
AWS Machine Learning Rushil Anirudh
Amazon shared a fix for a weird problem: fine-tuning Nova models on plain input-output data quietly kills their reasoning ability. Their trick, SDR, has the model generate its own chain-of-thought to train on, no humans or teacher models needed.
Fine-tuning a reasoning model sounds simple until you realize most of your training data doesn't have any reasoning in it. Someone just wrote 'question, answer' pairs, not 'question, step-by-step thinking, answer.' Amazon's applied science team ran into this with the Nova 2 family and found something ugly: when you train on those bare pairs with reasoning mode switched on, the model doesn't stay quiet about its thinking, it forgets how to think at all, even when you ask it to reason later at inference time.
The cause, they argue, is a loss function that only rewards output tokens. If there's no supervision on intermediate reasoning steps, the model learns to skip straight to the answer, a classic case of shortcut learning. And the cost is real. In one test on the LLaVA CoT dataset, keeping reasoning on during both training and inference lifted target performance by as much as 17 points over reasoning-off, once merging weights were tuned. Skip reasoning during fine-tuning, though, and that gain evaporates along with a chunk of general capability. Amazon's own numbers show math performance on a fine-tuned model cratering from 70 percent down to 6 percent.
The usual fix is model merging: blend the fine-tuned checkpoint back with the base model to recover lost skills. It works, sort of. But it's a trade-off machine, trading target-task gains for general capability, and you have to hand-tune the merge weight to find a livable middle ground. Amazon's alternative is what they call Self-Distilled Reasoning, or SDR. Instead of merging after the fact, they have the base Nova 2 Lite model generate its own chain-of-thought traces for every example in the training set, either by reasoning forward from just the question or backward from question-plus-answer using a hindsight-style prompt. Those traces get prepended to the outputs, and the model trains on the whole thing with reasoning turned on.
The results are the kind of clean comparison researchers like to show off. SDR matched or beat vanilla SFT on target task performance while recovering math accuracy to around 70 percent, statistically the same as the untouched base model, compared to merging's 68 percent, and without merging's forced trade-off between target and general skills. No teacher model, no human annotators, no post-hoc interpolation tuning. Just the model teaching itself using its own reasoning style, which conveniently sidesteps the annotation cost that made adding CoT traces impractical in the first place.
What's notable here isn't the specific numbers, it's the framing. Amazon is explicitly tying this to a broader pattern in the field: self-distillation as implicit regularization, keeping a fine-tuned model's policy tethered to its base model the way KL penalties do in RLHF. That's a more principled story than 'just merge the weights and hope,' and if it holds up across more domains, it could become a standard step in any reasoning-model fine-tuning pipeline, not just an Amazon-specific trick.
My take
This is a genuinely useful engineering fix dressed up as a research finding, and I don't say that as an insult. The real story is that reasoning-mode fine-tuning has a nasty failure mode almost nobody talks about outside the labs building these things, and Amazon found a cheap way around it instead of just recommending everyone buy more annotation. I'd like to see this tested outside Nova and against RFT-based approaches before calling it settled, but 'the model teaches itself using its own outputs' is going to keep showing up as the answer to overfitting problems, and that's worth watching.
Read more about this at: AWS Machine Learning
