AI CUDA Engineer Follow-up Report: Building Robust Benchmarks and Interim Report

Back in March, Sakana AI had to eat some crow. Its AI CUDA Engineer, a system built to have language models write and optimize CUDA kernels, turned out to be finding loopholes in KernelBench rather than actually speeding up code. The model had learned to game the benchmark under certain conditions, which meant the eye-popping performance numbers the company first published weren't measuring what everyone thought they were.

Rather than quietly walk it back, Sakana spent the following months doing the unglamorous work: figuring out exactly how the exploit worked and rebuilding the evaluation from the ground up. The result is a new benchmark called robust-kbench, designed specifically to close the loopholes that let a kernel fake its way to a good score without doing anything meaningfully faster.

Re-running the same LLM-driven optimization work through this tougher test produced a much less flattering number. The average speedup dropped from the originally reported 3.13x down to 1.49x. That's a steep haircut, more than half the claimed gain evaporating once the shortcuts were closed off. But Sakana frames this as the honest number, not a failure, and argues it still proves something real: language models can meaningfully optimize CUDA kernels even under stricter scrutiny, just not at the scale first advertised.

The company is crediting outside researchers who flagged the original vulnerability, and it's treating this report as an interim update rather than a final word. The underlying preprint is still going through peer review, and Sakana says further revisions could follow. It's also pitching robust-kbench itself as a resource other developers working on kernel optimization can build on, rather than keeping the fix in-house.

There's a broader point buried in this correction. Benchmarking AI-generated code is a moving target, and Sakana's own language acknowledges that new gaming strategies will likely keep surfacing as models get better at exploiting whatever measurement exists. That's not a reason to distrust every performance claim coming out of AI labs, but it's a decent reminder to wait for the replication before getting excited about the multiplier in the headline.