Inside the Megakernel Serving Engine for North Mini Code

Cohere has published a new serving engine for North Mini Code that leans on a decode megakernel, and the pitch is simple: stop treating decode like a parade of tiny GPU kernels. The code is on GitHub, and the company says the system is already good for OpenAI-compatible serving with tool calling, continuous batching, paged attention, and ragged sequence lengths. Point OpenCode at it, and it can code with it.

The numbers are the point. North Mini Code is a 30B model with 3.3B parameters active per token. In BF16, that means about 6.6 GB of weights streamed every decode step, plus roughly 0.5 GB of KV cache at 8K context. On an H100, with 3.35 TB/s of HBM bandwidth, the paper puts the speed-of-light ceiling at around 470 tok/s. vLLM hits 185 tok/s on this setup, which is only 39% of that ceiling.

Cohere’s megakernel gets to 292 tok/s at batch size 1, or 62% of the bandwidth limit, which works out to 1.58x faster than vLLM. The company says that margin holds across batch sizes and out to 256K of context, with no measurable loss of accuracy. That is a strong result for a serving engine, not just a demo, because most megakernel work has stopped at single-batch experiments.

The architecture is the interesting part. Instead of launching QKV, then attention, then MoE as separate kernels and forcing the GPU to wait at each boundary, the new engine runs the whole decode step as one persistent kernel. Each SM gets a task list, and dependencies are tracked with counters in global memory. That lets the engine shrink scheduling from whole operations down to tiles of operations, backfill idle SMs, drop false dependencies, and prefetch weights before the next activation is even ready.

Cohere says the implementation is just one CUDA file, with ordinary tiled GEMMs and paged attention reorganized to fit a single calling convention. No new compiler. No exotic abstraction. Just a lot less waiting around for the GPU to pretend it is busy.