The Sequence Knowledge #878: Beyond Transformer: What We Learned

For almost a decade, AI has basically been one big bet on self-attention. TheSequence just closed out an eight-part investigation into whether that bet is starting to look shaky, and the answer lands somewhere between "not yet" and "not forever."

The pitch for attention was never that it's the most elegant idea in machine learning. It's that every token can look at every other token, the math maps neatly onto a GPU, and the whole sequence trains in parallel. Throw more data and compute at it and the loss curve just keeps cooperating. That's a rare property, and it's why Transformers ate the field. But the bill comes due at scale: attention cost grows quadratically with sequence length, and the KV-cache that autoregressive decoding depends on grows linearly with every token generated. A 70B model chewing through long context can burn 40GB of VRAM just holding that cache. Past a million tokens, that's not a footnote anymore.

The series sorted the challengers into four camps. Recurrent models — the RNN revival and xLSTM, a modernized descendant of Hochreiter and Schmidhuber's 1990s design — trade attention's growing cache for a fixed-size hidden state, betting that clever reformulated math can make them parallelize during training after all. State space models, the Mamba lineage, are the most credible threat: they run as a convolution while training and a cheap recurrent scan during inference, though pure SSMs stumble on precise lookup, which is why the best versions mix in a handful of attention layers anyway. Text diffusion models like LLaDA, Gemini Diffusion, and Mercury throw out left-to-right decoding entirely, denoising a whole sequence in parallel instead — fast, but still chasing autoregressive models on raw quality. And liquid, continuous-time networks ditch the parallel-lookup framing altogether for something closer to adaptive dynamical systems, aiming for smaller, more efficient models rather than bigger ones.

None of these has actually beaten attention outright. What TheSequence argues instead is that the one-architecture era is ending, and what's coming is a patchwork: attention reserved for the moments where exact, lossless recall is worth paying quadratic costs for, and something linear running everywhere else. It's less a coronation of a new king and more an admission that the kingdom is getting messier.

Next up for the publication is a shift from architecture to compression — a series on knowledge distillation, the unglamorous process of squeezing a big expensive model's abilities into a smaller, cheaper one. It's a fitting sequel: if this series was about how models are built, the next one is about how they actually reach anyone who isn't sitting on a cluster of H100s.