​Sequential Attention: Making AI models leaner and faster without sacrificing accuracy

Feature selection sounds like a boring housekeeping task, but it's actually one of the thorniest problems in machine learning. Figuring out which inputs a model actually needs, and which ones are just noise, is technically NP-hard, meaning there's no fast way to solve it perfectly once you have a lot of variables. Deep neural networks make this worse because features interact in weird nonlinear ways. Something that looks useless on its own might become essential once paired with another feature, and vice versa.

Google Research's answer is called Sequential Attention, and the core idea is refreshingly simple even if the math underneath isn't. Instead of scoring every candidate feature at once like standard attention mechanisms do, it builds the subset one piece at a time. At each step it calculates attention scores for everything not yet chosen, grabs the highest-scoring one, locks it in, then recalculates scores for what's left. That recalculation is the clever part — it naturally accounts for redundancy, so the algorithm doesn't keep picking features that just repeat information it already has.

This matters because the traditional greedy approach to feature selection is brutally expensive. It typically means retraining or re-evaluating the model for every candidate at every step, which balloons cost fast on anything large. Google's researchers found they could fold this selection process directly into a single training run instead, cutting overhead dramatically. They also showed something neat mathematically: when applied to simple linear regression, Sequential Attention turns out to be equivalent to Orthogonal Matching Pursuit, a well-established algorithm with proven performance guarantees. That's a useful bit of validation for a method that otherwise might seem like a clever heuristic with no theoretical backbone.

The team has already extended the idea into SequentialAttention++, aimed at block sparsification — pruning entire chunks of weights rather than individual parameters, which actually speeds things up on real GPUs and TPUs rather than just looking good on paper. They've tested it on tasks like ImageNet classification and on the giant embedding tables used in recommender systems, where trimming down feature sets can mean real savings at scale. Google is now eyeing LLM pruning as the next target, hoping to strip out redundant attention heads and transformer blocks without tanking model quality, plus applications in genomics and drug discovery where picking the right handful of genetic features out of thousands is its own kind of nightmare.