Scaling Laws, Carefully

Lilian Weng · 3 weeks ago

Researchers studying scaling laws in deep learning have found that training loss decreases predictably as model size, dataset size, and compute increase following power-law relationships, with the Chinchilla paper (Hoffmann et al. 2022) challenging earlier findings from Kaplan et al. (2020) about optimal resource allocation. The Kaplan et al. study recommended allocating a 10x compute increase by scaling model size 5.5x but training tokens only 1.8x, while Chinchilla argued this approach leaves large models undertrained. These scaling laws enable practitioners to fit models on small experimental runs and extrapolate predictions for larger model training requirements.