​Sequential Attention: Making AI models leaner and faster without sacrificing accuracy

Google Research · 5 months ago

Researchers developed Sequential Attention, a technique for efficiently selecting the most important features, blocks, or components in neural networks during training rather than through expensive post-hoc analysis. The method uses attention scores to sequentially identify redundant components and achieved state-of-the-art results on standard benchmarks while maintaining accuracy. This approach enables building smaller, faster models and can be applied to feature selection, network pruning, large language models, and drug discovery without significantly increasing training costs.