GaLore: Advancing Large Model Training on Consumer-grade Hardware

Hugging Face Blog · 2 years ago

GaLore reduces memory requirements for training large language models by projecting gradients into lower-dimensional subspaces before optimizer processing. The technique achieves an 82.5% reduction in memory for optimizer states and enables training of 7-billion-parameter models on consumer GPUs like the NVIDIA RTX 4090. When combined with 8-bit quantization, GaLore allows researchers with limited computational resources to train larger models or use larger batch sizes on standard hardware.