REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

Apple Machine Learning Research · 1 month ago · 40

Researchers introduced REFACTOR-VLA to turn monolithic vision-language-action behavior into reusable typed motor programs learned with a wake/sleep loop. The system clusters motor-program segments in the sleep phase using a Behavioral-Equivalence Kernel driven by rollouts in a learned latent world model trained via a three-phase schedule, and it reports NMI scores for n=3 multi-seeding including 0.915 ± 0.013 on the Goal suite. Performance shifts as a bigger world model (188M to 430M parameters) worsens 4 out of 4 LIBERO benchmark suites while adding an auxiliary InfoNCE contrastive loss in Phase A improves the quality of skill clustering in Phase C.