How to Train Really Large Models on Many GPUs?
Lilian Weng
The article explains techniques for training large neural network models across multiple GPUs, including data parallelism, model parallelism, pipeline parallelism, tensor parallelism, and mixture-of-experts approaches. Key concrete details include GPipe's finding that bubble overhead becomes negligible when the number of microbatches exceeds 4 times the number of partitions (m > 4d), and PipeDream-2BW maintaining only two versions of model weights. These parallelism strategies enable distributed training by dividing computation, memory, and data across multiple workers to overcome single-GPU memory limitations and reduce training time.
Why it matters
[Updated on 2022-03-13: add expert choice routing.] [Updated on 2022-06-10]: Greg and I wrote a shorted and upgraded version of this post, published on OpenAI Blog: “Techniques for Training Large Neural Networks”
