Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate
Hugging Face
Hugging Face benchmarked ways to run the massive 176B-parameter BLOOM model fast on GPUs. DeepSpeed-Inference hit under 1 millisecond per token — brutally fast for a model this size.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
BLOOM is enormous. At 176 billion parameters, just holding the weights in bfloat16 eats 352GB, which is why Hugging Face's engineers reached for a full node of 8x80GB A100s as their baseline rig. Anything smaller and you're offloading to CPU or NVMe, which works but crawls in comparison.
The headline number here is startling: DeepSpeed-Inference, using tensor parallelism and custom fused CUDA kernels, generated tokens at 0.69 milliseconds each with a batch size of 128 on that 8-GPU setup. Push it to int8 quantization and you can still hit 0.71 milliseconds while using half the memory. Compare that to Accelerate, Hugging Face's own simpler library, which bottomed out around 10.89 milliseconds per token in bf16 before running out of memory at higher batch sizes. Both are legitimate approaches, but they're solving the problem very differently.
Accelerate uses naive pipeline parallelism — basically one GPU works while the other seven sit idle, then control passes down the line. It sounds wasteful, and it kind of is, but it's dead simple and works on almost any hardware configuration without modification. Users have run BLOOM on just two A100s with heavy offloading, though at 15 seconds per token, which is a different universe of slow. DeepSpeed-Inference instead splits tensors across all GPUs simultaneously, so every card is computing at once, at the cost of much heavier inter-GPU communication. That tradeoff — more coordination overhead for more parallel compute — is exactly why it wins on raw throughput.
There's a third option worth mentioning: DeepSpeed-ZeRO, which shards the model across GPUs and can, in principle, run entirely different input streams on each one. That means its effective throughput can be 8x or 16x higher than the raw per-GPU number suggests, once you're running separate generation jobs in parallel rather than one big batch. It's slower per-stream than the tensor-parallel approach, but the ability to fan out independent workloads is a genuinely different use case.
Loading time matters too, and it's easy to overlook. Deepspeed-Inference's pre-sharded checkpoints load in about a minute; the unsharded BLOOM checkpoint takes over 10 minutes. Accelerate lands in the middle at roughly two minutes. None of this matters once the model is warm and serving requests continuously, but for anyone spinning up and tearing down inference jobs, it adds up fast.
My take — AI-written commentary, not fact-checked reporting
What strikes me is how much performance is left on the table by default tooling — going from Accelerate's naive pipeline to DeepSpeed's custom kernels is a 15x+ speedup on the same hardware, no model changes required. That gap says more about how immature large-model serving infrastructure still is than about BLOOM itself. If you care about open models actually being usable outside a handful of labs with A100 clusters, this kind of unglamorous kernel engineering matters more than the next capability jump.
Read more about this at: Hugging Face