Scaling Laws, Carefully
Lilian Weng
Researchers studying scaling laws in deep learning have found that training loss decreases predictably as model size, dataset size, and compute increase following power-law relationships, with the Chinchilla paper (Hoffmann et al. 2022) challenging earlier findings from Kaplan et al. (2020) about optimal resource allocation. The Kaplan et al. study recommended allocating a 10x compute increase by scaling model size 5.5x but training tokens only 1.8x, while Chinchilla argued this approach leaves large models undertrained. These scaling laws enable practitioners to fit models on small experimental runs and extrapolate predictions for larger model training requirements.
Why it matters
Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between $N$ and $D$.
