Preparing data for supervised fine-tuning Part 2: Advanced data strategies
Amazon Web Services 1 month ago 30 ● 2 sources
The article lays out advanced strategies to improve supervised fine-tuning datasets after formatting and quality checks. It recommends planning for roughly 2,000 high-quality training samples as a starting point, with learning-curve analysis using checkpoints saved every 10–20 percent of training to find when gains saturate. As a result, teams can decide when to stop, curate a smaller higher-value subset, use augmentation (including synthetic reasoning traces) to expand coverage, and apply controlled data mixing to reduce catastrophic forgetting.