Preparing data for supervised fine-tuning Part 2: Advanced data strategies
Amazon Web Services Krishnateja Killamsetty ● Covered by 2 sources
AWS says SFT tuning doesn’t stop at clean data; the real work is deciding how much, which parts, and when to mix in more. The surprise: a smaller, better subset can beat a much bigger pile of examples.
Based on reporting by Amazon Web Services, Krishnateja Killamsetty — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Cleaning and formatting a supervised fine-tuning dataset is only the first gate. AWS’s second post in its SFT series moves past the basics and into the awkward questions that decide whether a model improves or just burns compute: how much data is enough, when a subset beats the full set, how augmentation should be verified, and when mixing data protects a model’s general skills.
The first checkpoint is simple to say and annoying to do: build a real evaluation benchmark before you chase bigger datasets. AWS suggests treating about 2,000 high-quality training samples as a rough starting point for a typical SFT task, but the right number can swing widely. A simple style change may land with 500 examples. A complex multi-step reasoning task may need 10,000 or more. The point is not the number itself. It’s figuring out where your curve flattens.
That’s where learning-curve analysis comes in. Train once on the full dataset, save checkpoints every 10–20 percent of training, and evaluate each one on held-out data. If doubling the amount of data improves the main metric by less than 1 to 2 percentage points, AWS treats that as the saturation point. At that stage, more of the same is mostly noise. The source points to work showing that SFT does not follow the same neat scaling pattern as pretraining. One cited result found that 128 epochs on 400 reasoning examples beat a single epoch on 51,200 examples by 12–26 percentage points on AIME and GPQA.
Once the curve starts flattening, subset selection can do more for you than brute force. Methods such as DEITA, DELIFT, and coreset selection aim to keep the smallest set that still covers the task space. AWS cites AlpaGasus, where filtering to the top 20 percent by quality trained faster and scored higher than the full set. The payoff is not just speed. Fewer low-value examples also mean fewer updates nudging the model away from what it already knew.
Augmentation and mixing are the other two levers. AWS says the most useful augmentation for modern SFT is synthetic reasoning traces and demonstrations, whether distilled from a stronger model, self-generated and filtered, or based on expert-written examples that are paraphrased into multiple styles. But every synthetic example still has to clear the same quality bar as human data. Mixing, meanwhile, is less about making the target task better and more about keeping the model from forgetting everything else. The warning is blunt: a tiny general-data mix can dominate by token count if those sequences are much longer, so token ratios matter more than sample ratios.
My take — AI-written commentary, not fact-checked reporting
The cleanest takeaway here is that SFT is becoming less like data hoarding and more like data editing. That’s a good thing, because the internet already has enough junk in it without turning fine-tuning into a landfill with a learning rate. The industry keeps pretending scale is the whole story; AWS is basically saying no, precision still matters, and the math is getting less forgiving about sloppy habits.
Read more about this at: Amazon Web Services