Sparser, Faster, Lighter Transformer Language Models

Sakana AI

Sakana AI and NVIDIA developed new GPU kernels and data formats to accelerate sparse transformer language models by reshaping sparsity patterns to match hardware capabilities rather than forcing hardware adaptation. The hybrid sparsity format (TwELL) achieved over 20% speedups and significant memory and energy savings in billion-parameter scale models. This enables more efficient inference and training of large language models by better exploiting the natural sparsity that emerges in transformer feedforward layers.