How to Train Really Large Models on Many GPUs?

No single GPU can hold a modern frontier model. That's the blunt fact driving this entire field of engineering, and Lilian Weng's deep dive lays out just how many different tricks researchers have stacked on top of each other to get around it. Weights alone can run into the tens of billions of numbers, and then you need room for gradients and optimizer states on top of that. Add a training corpus that would take a single machine years to chew through, and parallelism stops being optional.

The simplest fix is data parallelism: copy the whole model onto every worker, hand each one a slice of the data, then sync up. The catch is deciding how strict that sync has to be. Bulk synchronous parallel setups wait for everyone before moving on, which keeps training stable but wastes time on stragglers. Asynchronous setups skip the waiting and pay for it with stale weights. PyTorch's answer, gradient accumulation in DDP, splits the difference by syncing every few iterations instead of every single one, and bucketing gradients into batched AllReduce calls to squeeze out more throughput.

When the model itself is too big for one card, you have to cut it up, and that's where pipeline parallelism earns its keep. A naive vertical split just creates idle GPUs standing around waiting on each other. Google's GPipe attacked this by chopping each minibatch into smaller microbatches so stages can overlap, shrinking the wasted "bubble" time down to something close to negligible once you run enough microbatches. Microsoft's PipeDream took a different route with one-forward-one-backward scheduling, but that opened a messier problem: which version of the weights should a given microbatch actually use. Its fix, stashing multiple weight versions and later trimming that down to just two with PipeDream-2BW, is a good example of how these systems trade memory for speed in very deliberate ways.

Tensor parallelism goes a level deeper, splitting individual matrix multiplications rather than whole layers. Nvidia's Megatron-LM does this inside transformer MLP and attention blocks, chopping weight matrices by column so chunks of the same computation run on different devices simultaneously. Combine that with pipeline and data parallelism, as Nvidia's later PTD-P work does by scattering non-contiguous chunks of layers across workers, and you get meaningfully smaller bubbles than a plain GPipe schedule.

Then there's mixture-of-experts, which sidesteps the problem entirely by not running the whole model on every input. A gating network learns to route each input to only a handful of expert sub-networks, so you get a model with a huge total parameter count without paying the full compute cost per token. Google's original sparsely-gated MoE work added noise and a load-balancing loss specifically so the gate doesn't just keep favoring the same few experts, which is the kind of unglamorous fix that makes the whole approach usable at scale rather than just clever on paper.