[AINews] Megakernels are so dead and so back

There's a fun contradiction buried in yesterday's Latent Space podcast: engineers spent a chunk of the episode declaring megakernels dead, then spent the rest of the newsletter covering Cursor's team shipping a new one.

The case against megakernels, laid out by Ali on the show, is pretty simple once you strip the jargon. The whole pitch for fusing an entire forward pass into one giant hand-written kernel was to kill launch overhead and stop GPUs from sitting idle between operations. But tensor parallelism breaks that dream fast — if half your attention matrix lives on one GPU and half on another, you still need the GPUs to talk to each other before you can run softmax. No amount of fusion avoids that handshake. And writing a 67,000-line kernel that squeezes out marginal gains, while TensorRT-LLM and modular kernel libraries let you optimize and parallelize each piece separately, just isn't worth it for anyone running production inference at scale. Ali's read: it was a legitimate research direction, but the complexity cost never paid off outside of research shops. Then there's Nvidia's Rubin architecture, which reportedly bakes in dependency-trigger scheduling — the fix for straggler CTAs that used to be megakernels' main justification. If kernel two can now launch as soon as seven of kernel one's ten thread blocks finish, instead of waiting for stragglers, a big chunk of the fusion argument evaporates at the hardware level.

And yet, on the very same day this eulogy aired, Stuart Sul — one of the original ThunderKittens megakernel coauthors — helped release Mixture of Kittens, a megakernel built for Cursor. It's not a toy: Cursor claims a 41% jump in tokens per second, which at serving scale is worth real money, not a rounding error. Separately, the newsletter notes Cursor open-sourced MoK, an NVL72 MoE training megakernel, with some of the most concrete performance numbers in the whole training-systems space this week.

So the honest takeaway isn't that megakernels vanished. It's that the calculus shifted. Hardware vendors are closing the gap that once justified extreme hand-fusion, and most serious inference providers have moved on to modular, per-component optimization instead. But for teams with the engineering budget and a narrow enough target — a specific MoE architecture, a specific training regime — a well-built megakernel can still beat the generic stack. The technique didn't die so much as it got demoted from general-purpose necessity to specialist's tool, which is a much less dramatic story but probably the truer one.

That tension also mirrors the rest of the day's AI news: DeepSeek-V4-Flash pricing pressure reshaping routing decisions, Not Diamond Code picking models per coding step, harness design outperforming raw model swaps by 5-30x on cost. Everywhere you look, the industry is optimizing at the margins because the big architectural wins are getting harder to find.