The Technology Behind BLOOM Training

Hugging Face Blog · 4 years ago

Researchers trained BLOOM, a 176 billion parameter language model, using 384 NVIDIA A100 GPUs on France's Jean Zay supercomputer with a custom Megatron-DeepSpeed software framework. The training took 3.5 months to complete between March and July 2022, processing 350 billion tokens across 59 languages. The project demonstrated that large-scale multilingual model training could be accomplished by a distributed team using open-source software and published infrastructure, making the approach and final model publicly accessible.