REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs
Apple Machine Learning Research
Researchers introduced REFACTOR-VLA to turn monolithic vision-language-action behavior into reusable typed motor programs learned with a wake/sleep loop. The system clusters motor-program segments in the sleep phase using a Behavioral-Equivalence Kernel driven by rollouts in a learned latent world model trained via a three-phase schedule, and it reports NMI scores for n=3 multi-seeding including 0.915 ± 0.013 on the Goal suite. Performance shifts as a bigger world model (188M to 430M parameters) worsens 4 out of 4 LIBERO benchmark suites while adding an auxiliary InfoNCE contrastive loss in Phase A improves the quality of skill clustering in Phase C.
Why it matters
Most current vision-language-action (VLA) models—such as OpenVLA, π0, RT-2, and RDT-1B—are “monolithic.” This means they generate raw motor commands or very short sequences of actions, without organizing behaviors into reusable, well-defined abstractions. As a result, these models perform poorly on long-horizon (multi-step) tasks, and it’s difficult to interpret what they have learned. Existing approaches for discovering skills often avoid the core problem of deciding when two action sequences are “behaviorally equivalent.” For example, AtomicVLA and AtomSkill group action sequences by…