ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

Together AI · 3 weeks ago

ParallelKernelBench is a new benchmark for evaluating whether large language models can write optimized multi-GPU CUDA kernels, testing frontier models like GPT-5.5 and Gemini 3 Pro on 87 real-world problems from production codebases. The best model solved only 28 of 87 problems in zero-shot evaluation, with just 22 producing kernels faster than the PyTorch + NCCL baseline, and even with three sampling attempts the success rate peaked at 31%. Despite the poor overall performance, some generated kernels exceeded publicly available implementations, including a novel kernel for NVIDIA NeMo-RL's GRPO training loop, suggesting LLMs struggle with rank coordination and communication optimization but occasionally produce genuinely useful new code.