Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel

Hugging Face Blog · 4 years ago

PyTorch's Fully Sharded Data Parallel (FSDP) feature, integrated into the Accelerate library, enables training of large models by distributing parameters and gradients across multiple GPUs without requiring code changes. On two 24GB GPUs, FSDP allowed batch sizes of 22 for the 762M-parameter GPT-2 Large model compared to batch size 7 with standard distributed data parallel, and successfully trained the 1.5B-parameter GPT-2 XL model with batch size 14 when using CPU offloading, whereas standard methods ran out of memory. Users can now train large models on minimal hardware by adjusting the minimum parameter threshold for layer wrapping and enabling CPU offloading, though they must prepare the model before creating the optimizer to avoid memory waste.