How to Train Really Large Models on Many GPUs?

Lilian Weng · 4 years ago

The article explains techniques for training large neural network models across multiple GPUs, including data parallelism, model parallelism, pipeline parallelism, tensor parallelism, and mixture-of-experts approaches. Key concrete details include GPipe's finding that bubble overhead becomes negligible when the number of microbatches exceeds 4 times the number of partitions (m > 4d), and PipeDream-2BW maintaining only two versions of model weights. These parallelism strategies enable distributed training by dividing computation, memory, and data across multiple workers to overcome single-GPU memory limitations and reduce training time.