PROOF-Gen: From Optimized Data to Better Distillation
Apple Machine Learning Research
Tool-calling distillation pipelines that repeatedly re-run supervised fine-tuning on teacher-generated trajectories keep paying the frontier-teacher cost while generating-and-filtering only passing cases. On τ 2-bench, 57% of teacher trials fail, with two-thirds of those failures being near-misses, so the current process leaves the hardest scenarios without useful training signal. PROOF-Gen changes the post-training approach by turning those optimized data signals (including near-misses) into better distillation for deployable models.
Why it matters
Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models. Post-training pipelines that drive shipped tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle, yet the mechanism is generate-and-filter (keep the teacher’s passing trajectories, discard the rest) and each cycle leaves behind the same hard scenarios because failures supply no signal. On τ 2-bench, 57% of teacher trials fail, two-thirds of them near-misses (most tool calls correct, undone…