Unlocking asynchronicity in continuous batching

Renting an H200 GPU costs about five bucks an hour, which sounds cheap until you run one for a full day and the bill hits $120. So if you're paying for that hardware, you want it computing, not twiddling its thumbs. Hugging Face's engineering team found that a shocking amount of GPU time in LLM inference is exactly that: idle, waiting on the CPU to catch up.

The culprit is how continuous batching normally works. The CPU picks which requests go into the next batch, updates the KV cache, evicts finished requests, and hands the data to the GPU. The GPU crunches numbers, then hands results back. The problem is this happens in strict turns — while the GPU computes, the CPU sits idle, and while the CPU reshuffles the batch, the GPU sits idle. Neither is ever doing useful work at the same moment. Profiling an 8B model generating 8,000 tokens with a batch size of 32 showed this synchronous dance ate up 24% of the 300.6-second run — nearly a quarter of total time spent doing nothing productive on the GPU side.

The fix Hugging Face lays out doesn't touch the model or require new kernels. It's purely about hardware choreography, using CUDA streams instead of PyTorch's default synchronizing stream. Streams let you queue operations that run independently and in parallel, rather than everything blocking on a single sequential timeline. By assigning separate streams to host-to-device transfers, GPU compute, and device-to-host transfers, the CPU can fire off work and immediately move on instead of waiting around.

But streams alone create chaos without ordering — early attempts produced results instantly, and wrong, because compute started before data even arrived. That's where CUDA events come in: markers that force one stream to wait for another to finish a specific step, without blocking the CPU or unrelated streams. Insert an event after the transfer, make compute wait on it, insert another after compute, make the output transfer wait on that. Now the GPU enforces correct sequencing on its own, and the CPU is free the moment it's dispatched everything.

That freed-up CPU window is the real prize. Instead of standing by, the CPU can start preparing batch N+1 while batch N is still crunching on the GPU — provided it carefully avoids overwriting buffers still in use and correctly threads newly sampled tokens into the next batch's inputs. Hugging Face says this asynchronous approach has already been built into the transformers library's continuous batching implementation, following the same steps outlined in this post. The theoretical payoff: shaving that 300-second run down toward 228 seconds, a 24% speedup with zero changes to the model itself — just smarter use of the hardware you're already renting.