The Technology Behind BLOOM Training
Hugging Face Blog ● Covered by 2 sources
Researchers trained BLOOM, a 176 billion parameter language model, using 384 NVIDIA A100 GPUs on France's Jean Zay supercomputer with a custom Megatron-DeepSpeed software framework. The training took 3.5 months to complete between March and July 2022, processing 350 billion tokens across 59 languages. The project demonstrated that large-scale multilingual model training could be accomplished by a distributed team using open-source software and published infrastructure, making the approach and final model publicly accessible.
