RayRoPE: Projective Ray Positional Encoding for Multi-View Attention
Apple ML Research
Researchers introduced RayRoPE, a positional encoding method for multi-view transformers that represents patch positions using predicted 3D points along camera rays rather than ray directions. The method achieves 15% relative improvement on LPIPS metrics in the CO3D dataset for novel-view synthesis compared to alternative position encoding schemes. RayRoPE enables geometry-aware attention that maintains SE(3) invariance and can incorporate RGB-D inputs, improving performance on multi-view 3D reconstruction tasks.
Why it matters
We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can be adaptive to the geometry of the underlying scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet the above desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays but leverages a predicted point along the ray instead of the direction for a…
