The Technology Behind BLOOM Training
Hugging Face ● Covered by 2 sources
Hugging Face just laid out exactly how it built BLOOM, its 176B-parameter multilingual model. Turns out training a giant open model takes 384 A100s, 3.5 months, and a small army of borrowed expertise.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Everyone loves to argue about whether big language models should be open, but almost nobody talks about the plumbing that makes training one possible in the first place. Hugging Face's new writeup on BLOOM fixes that, and it reads less like a marketing post and more like an engineering diary from a team that spent 14 months wrestling hardware, software and a supercomputer into submission.
The raw numbers are the easy part to grasp. BLOOM trained on 384 NVIDIA A100 80GB GPUs, spread across 48 nodes on Jean Zay, the French government's supercomputer run by GENCI and IDRIS. Add another 32 spare GPUs for when things inevitably broke. The run consumed roughly 1 million compute hours between March and July 2022, chewing through 350 billion tokens pulled from 1.5TB of cleaned text across 46 languages. A single full checkpoint, weights plus optimizer states, weighs in at 2.3TB. Even just the bf16 weights alone are 329GB. This is not a laptop project.
What's more interesting than the scale is the software stitched together to make it work: Megatron-DeepSpeed, a mashup of NVIDIA's Megatron-LM and Microsoft's DeepSpeed. The team used a technique called 3D parallelism, combining three distinct tricks. Tensor parallelism, borrowed from Megatron-LM, splits individual weight matrices across GPUs so no single chip has to hold an entire tensor. Pipeline parallelism slices the model vertically by layer and staggers micro-batches through it like an assembly line, which cuts down the idle
My take — AI-written commentary, not fact-checked reporting
What strikes me most is that this wasn't Google or OpenAI flexing compute — it was a scrappy BigScience crew borrowing a French government supercomputer and stitching together open tools from Microsoft, NVIDIA and PyTorch to pull off something genuinely hard. That's the open-model playbook working exactly as it should: pool resources, share the engineering notes, and let anyone reproduce the recipe instead of guessing at it from a paper's acknowledgments section.
Read more about this at: Hugging Face