Mamba Explained
The Gradient Kola Ayonrinde
Mamba is a new AI architecture that ditches Transformer-style attention for something called a Selective State Space Model. It matches Transformer performance but runs up to 5x faster and doesn't choke on huge contexts.
Based on reporting by The Gradient, Kola Ayonrinde — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
For the last few years, basically every headline AI breakthrough has had the same engine under the hood: the Transformer, powered by attention. Mamba is one of the first serious challengers to that dominance, and it comes from a different lineage entirely — State Space Models, a idea borrowed from control theory rather than the attention playbook that OpenAI and Google popularized.
The pitch is simple to state and hard to pull off. Attention lets every token look back at every previous token, which is great for accuracy but brutal for math: it costs O(n²) time to train and O(n) memory just to store the key-value cache. Push the context window toward a million tokens and things slow to a crawl or blow out your GPU memory. Mamba's authors, Albert Gu and Tri Dao, went after that quadratic bottleneck directly, replacing attention with a state space mechanism that scales linearly instead. The result, according to their benchmarks on The Pile, is a 3-billion-parameter Mamba model that beats same-sized Transformers and matches ones twice its size — while running up to five times faster at inference.
The trick isn't just using an SSM, which is an old idea, but making it selective. Vanilla state space models apply the same fixed transformation to every token, like an assembly line that treats every part identically regardless of what it actually is. That's efficient but dumb — it can't decide what to forget or remember based on context. Mamba fixes this by making its core matrices functions of the input itself, so the model dynamically decides, token by token, what's worth carrying forward into its hidden state and what's safe to discard. Gu and Dao frame this as the real design principle sequence models need: small state for efficiency, but selective enough to keep whatever actually matters from the context.
Think of it as the difference between a Transformer's near-photographic memory, which stores everything and figures out what's relevant when it needs it, and a classic RNN, which forgets almost everything and just keeps a tiny compressed summary. Mamba tries to land in between: an RNN-sized memory footprint with enough selectivity to behave more like attention when it counts. Whether that tradeoff holds up at GPT-4 scale is still an open question, but the early numbers suggest state space models deserve a real seat at the table alongside Transformers, not just a footnote.
My take — AI-written commentary, not fact-checked reporting
I think the Transformer's reign has made people lazy about assuming attention is the only path forward, and Mamba is a healthy reminder that architecture research didn't stop in 2017. I'm rooting for anything that makes million-token context cheap and open enough that smaller labs can actually run it, because right now that kind of scale is a rich-lab-only luxury. Watch this space — if selective SSMs scale the way early results suggest, the next wave of open models might not be Transformers at all.
Read more about this at: The Gradient