VideoSEMA: a scalable and efficient Mamba-like attention for video understanding

arXiv cs.AI · 6 hours ago

Researchers introduced VideoSEMA, a video classification model combining space-time attention mechanisms with a Mamba-like architecture that uses local window and global averaging operations. On Kinetics-400 benchmark, VideoSEMA outperformed heavier vision transformers and Mamba models while maintaining competitive parameter counts on Something-Something-v2, and showed more graceful accuracy degradation when scaling image resolution from 224² to 1024² without fine-tuning. The approach reduces computational cost while maintaining comparable performance by proving mathematical equivalence between split and full space-time attention under certain rank conditions.