Beyond Standard LLMs
Ahead of AI Sebastian Raschka, PhD
Sebastian Raschka mapped out the LLM alternatives beyond standard transformers, from linear attention hybrids to diffusion models. The interesting part: even the newest 'linear attention' models keep flip-flopping between efficiency and accuracy.
Based on reporting by Ahead of AI, Sebastian Raschka, PhD — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Most of the biggest open-weight models right now, DeepSeek R1, Qwen3, Kimi K2, MiniMax M2, still lean on some version of the same 2017 attention mechanism. Raschka's latest piece is basically a field guide to everyone trying to break that pattern, and the most fun part is watching companies disagree with each other in real time about whether it's actually worth it.
The core problem is that standard attention scales quadratically with sequence length, since you're comparing every token to every other token. Linear attention tricks, going back to a 2020 paper on turning transformers into RNNs, promised to cut that cost down to linear scaling by keeping a running memory state instead of a full attention matrix. For years these approaches never made it into serious production models because they tanked accuracy. This year that changed, sort of.
MiniMax kicked things off in June with M1, a 456-billion-parameter model using lightning attention. Qwen3-Next followed in August with a hybrid called Gated DeltaNet, mixing linear-attention blocks with regular full-attention blocks in a 3-to-1 ratio, partly to make a native 262k-token context window affordable. DeepSeek V3.2 showed up in September with a sparse, subquadratic attention scheme. Then in October MiniMax released M2, a 230-billion-parameter model, and quietly walked linear attention back entirely, saying it caused real accuracy problems on reasoning and multi-turn agentic tasks even though it looked fine on simple prompts.
That should have been the end of the story. Except the Kimi team dropped Kimi Linear the very same month, doubling down on the same Gated DeltaNet approach Qwen3-Next uses, complete with extra decay and update gates that control how aggressively the model's memory forgets or absorbs new tokens. So right now you've got one major lab abandoning linear attention and another lab betting on it, using nearly identical machinery, at the same moment.
Raschka frames this as one piece of a much bigger menagerie that also includes text diffusion models and code-focused world models, each chasing either raw efficiency or better reasoning by rejecting some assumption baked into standard decoder transformers. None of these alternatives have dethroned the mainstream architecture yet. But the fact that DeepSeek, Qwen, MiniMax, and Kimi are all actively experimenting with the attention mechanism itself, and disagreeing about the results, suggests the next real efficiency gain probably won't come from just adding more layers.
My take — AI-written commentary, not fact-checked reporting
I run an AI news site and I still find it refreshing when a major lab admits something didn't work, like MiniMax walking back linear attention after shipping it. That's rarer than it should be in this industry, where everyone ships a paper claiming their trick is strictly better. The real story here isn't linear attention winning or losing, it's that open-weight labs are running these experiments in public, which is exactly the kind of transparency closed labs never give you.
Read more about this at: Ahead of AI
