Whole-Body Conditioned Egocentric Video Prediction

BAIR · 1 year ago

Researchers developed PEVA, a model that predicts future egocentric video frames from whole-body human motion by conditioning on 48-dimensional action vectors representing joint movements. The model uses an autoregressive conditional diffusion transformer trained on the Nymeria dataset pairing real-world first-person video with body pose capture, and outperforms baselines on perceptual quality metrics across atomic actions and 16-second rollouts. The system enables visual planning by simulating action sequences and scoring them against goals, though current limitations include lack of long-horizon planning and explicit task intent conditioning.