[AINews] Megakernels are so dead and so back
Latent Space ● Covered by 2 sources
Megakernels — those giant hand-fused GPU kernels — got declared dead in a podcast debate, then Cursor open-sourced one anyway. Turns out research 'dead ends' still ship real speedups: this one claims a 41% tokens-per-second boost.
Based on reporting by Latent Space — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
There's a fun contradiction buried in yesterday's AI News roundup. On one hand, engineers on Latent Space's Inference Engineering Masterclass pod spent a good chunk of the episode burying megakernels — those sprawling, hand-fused GPU kernels that try to eliminate launch overhead by stitching an entire forward pass into one giant blob of code. On the other hand, Cursor turned around and open-sourced a megakernel called Mixture of Kittens the very same day, claiming a 41% jump in overall tokens per second.
The podcast argument, voiced mainly by Ali, is pretty concrete. With tensor parallelism, a matrix gets split across GPUs, and any nonlinear operation — softmax, exponentiation, whatever — needs the full row before it can proceed. That means GPUs have to talk to each other regardless of how fused your kernel is, because you can't dodge the communication step just by writing cleverer code. Ali also points out that megakernels are brutally hard to optimize well, and that people he's talked to at companies who built fused megakernels often quietly ship TensorRT-LLM or modular kernels in production instead, because separately optimized components that parallelize with each other end up faster.
Then there's the hardware angle. One NVIDIA tech lead, in a Twitter thread about Rubin's specs, described dependency triggers — a mechanism where if kernel two needs ten CTAs and kernel one has finished seven with three stragglers, kernel two can just launch with the seven that are ready. That's the straggler-CTA problem megakernels were partly built to route around, and if Rubin bakes the fix into silicon, the argument goes, an entire research direction loses its reason for existing. Nobody serious is running a 67,000-line hand-fused forward pass kernel in production, the podcast argues — the people doing it are doing it for research, full stop.
And yet Mixture of Kittens exists, built by Stuart Sul, a coauthor on Ben Spector's original megakernel work (the name nods to ThunderKittens, from Dan Fu's group). It's an NVL72 MoE training megakernel, and Cursor is putting a real number behind it: 41% more tokens per second overall, which at scale is the kind of gain that translates into serious money. So the
My take — AI-written commentary, not fact-checked reporting
Calling a whole research direction dead while a team ships a working megakernel with a 41% speedup the same week is a good reminder that engineering trends rarely die cleanly — they just get niche, then someone with the right incentives makes the niche pay off. Betting everything on Rubin's hardware fix arriving before that happens seems premature; the people building the workaround aren't waiting for permission.
Read more about this at: Latent Space