Sparser, Faster, Lighter Transformer Language Models
Sakana AI
Sakana AI and NVIDIA built new GPU kernels that make sparse transformer LLMs actually run faster instead of slower. Turns out reshaping sparsity to fit the hardware beats forcing hardware to handle messy sparsity.
Based on reporting by Sakana AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
There's a weird trap in deep learning that engineers have quietly hated for years: teaching a model to skip unnecessary computation often ends up making it slower, not faster. Sakana AI, working with NVIDIA, just published a fix aimed squarely at that problem, and they're presenting it at ICML 2026.
The root issue is that language models are naturally sparse. In feedforward layers, more than 95% of neurons stay dark for any given token, much like how your brain only fires the neurons relevant to a specific thought. GPUs, though, are built to crunch dense, uniform blocks of numbers. When a model tries to skip inactive neurons, the resulting memory access pattern turns ragged and unpredictable, and the chip chokes on it. So sparsity, which should save compute, instead adds overhead.
Instead of trying to make GPUs tolerate irregular sparsity, Sakana flipped the problem. They designed a new data format called TwELL that reshapes sparse computation into something GPUs are already good at handling. Most tokens, something like 99% of them, are extremely sparse and get routed down a streamlined fast path. The rare tokens that stay dense get shunted into a separate backup matrix, so the system doesn't have to slow everything down just to accommodate outliers.
They tested this on billion-parameter models, both training and inference, and reported speedups above 20%, plus even bigger reductions in peak memory usage and energy consumption. That's not a marginal tweak. At the scale frontier labs operate on, a 20% inference speedup translates into real dollars and real carbon.
The kernels and format are open-source, which matters given who's behind this. NVIDIA has obvious incentive to make sparse models run efficiently on its own hardware, and Sakana gets to show off systems-level chops beyond its usual model-building work. Whether other hardware vendors adopt similar tricks, or whether this stays an NVIDIA-flavored optimization, is the open question worth watching.
My take — AI-written commentary, not fact-checked reporting
This is the unglamorous infrastructure work that actually moves the needle, more than another benchmark-chasing model release ever could. I'm a sucker for efficiency gains that come from rethinking the hardware-software boundary rather than just throwing more GPUs at the problem, and a 20%+ speedup with lower energy use is the kind of thing that should get more attention than it will. Open-sourcing the kernels is the right call too — if this becomes the default way sparse transformers get built, credit should go where it's due.
Read more about this at: Sakana AI
