PROOF-Gen: From Optimized Data to Better Distillation
Apple Machine Learning Research
Tool-calling distillation pipelines that repeatedly re-run supervised fine-tuning on teacher-generated trajectories keep paying the frontier-teacher cost while generating-and-filtering only passing cases. On τ 2-bench, 57% of teacher trials fail, with two-thirds of those failures being near-misses, so the current process leaves the hardest scenarios without useful training signal. PROOF-Gen changes the post-training approach by turning those optimized data signals (including near-misses) into better distillation for deployable models.
Why it matters
Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models. Post-training pipelines that drive shipped tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle, yet the mechanism is generate-and-filter (keep the teacher’s passing trajectories, discard the rest) and each cycle leaves behind the same hard scenarios because failures supply no signal. On τ 2-bench, 57% of teacher trials fail, two-thirds of them near-misses (most tool calls correct, undone…
Related stories
Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation
Apple Machine Learning Research · 1 month ago ·
37
The Sequence Knowledge - 928: The Missing 5%: Why Distillation Is Harder Than It Looks
TheSequence · 1 month ago ·
44
The Sequence Knowledge #886: Demystifying Model Distillation
Substack · 3 months ago ·
9