Visualize and understand GPU memory in PyTorch

Hugging Face Blog · 1 year ago

PyTorch provides a built-in tool (`torch.cuda.memory._record_memory_history()`) that generates visual profiles of GPU memory usage during model training. The memory profile for a 1.5B parameter model training loop shows three distinct spikes, each corresponding to a training iteration with peaks occurring during forward passes, backward passes, and optimizer steps. Total GPU memory required equals model parameters plus optimizer state plus the maximum of either (gradients plus optimizer intermediates) or (activations), and can be estimated using formulas based on parameter count, batch size, and sequence length.