Accelerate Large Model Training using DeepSpeed

Hugging Face Blog · 4 years ago

The Accelerate library enables training of large models using DeepSpeed's ZeRO optimization technique, which distributes optimizer states, gradients, and model parameters across GPUs and CPU memory. In a benchmark with a 900M-parameter DeBERTa model, DeepSpeed ZeRO Stage-2 achieved a batch size of 40 compared to DDP's maximum of 8, resulting in 3.5X faster training time without performance loss. This allows users to fit larger models and batch sizes on existing hardware without modifying their training code.