Unlocking asynchronicity in continuous batching

Hugging Face Blog · 2 months ago

Researchers developed asynchronous batching to eliminate idle time in language model inference by running CPU batch preparation and GPU computation in parallel instead of sequentially. Profiling an 8B model generating 8,000 tokens showed that 24% of total runtime was wasted waiting, which asynchronous batching could theoretically reduce to 228 seconds from 300 seconds. The technique uses separate CUDA streams for data transfers and computation, with GPU events enforcing synchronization order so the CPU can prepare the next batch while the current batch computes.