Scaling Laws for Mixture Pretraining Under Data Constraints
Apple Machine Learning Research
Mixture pretraining studies quantified how mixing scarce target-domain data with abundant generic data affects target-domain performance. Repetition of scarce target corpora can be reused 15–20 times, with the best repetition level depending on target data size, compute budget, and model scale. The work introduces a repetition-aware mixture scaling law so mixture configurations can be chosen systematically instead of by trial and error, changing how data mixtures are set under data constraints.
Why it matters
As language models scale, the amount of data they require grows – yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade-off: too little target data in the mixture underexposes the model to the target domain, while too much target data repeats the same examples excessively, yielding diminishing returns and eventual overfitting. We study this trade-off across more than 2,000 language-model training runs…