Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate

Hugging Face · 3 years ago · 1

Researchers demonstrated inference optimizations for the 176-billion-parameter BLOOM model using DeepSpeed and Hugging Face Accelerate across multiple GPU configurations. DeepSpeed-Inference achieved sub-1 millisecond per-token throughput at batch size 128 on 8x80GB A100 GPUs, compared to 0.69 milliseconds in the benchmark, while Accelerate reached 10.89 milliseconds at batch size 32 on the same hardware. The results show that tensor parallelism with fused CUDA kernels outperforms pipeline parallelism by keeping all GPUs active during computation, enabling larger batch sizes and faster overall generation speeds.