Mamba Explained

The Gradient · 2 years ago

Mamba is a State Space Model architecture that replaces the Attention mechanism in Transformers with a control-theory-inspired SSM while maintaining similar performance and scaling properties. A Mamba-3B model outperforms same-sized Transformers and matches Transformers twice its size, while running up to 5 times faster and handling sequence lengths up to 1 million tokens. This enables faster inference and longer context windows by eliminating the quadratic time complexity bottleneck that limits Transformer efficiency.