How AI training scales

OpenAI Blog · 7 years ago

Researchers found that gradient noise scale, a statistical measure, predicts how well neural network training can be parallelized across different tasks. Complex tasks generate noisier gradients, which means larger batch sizes will become practical as AI systems grow, potentially removing a scaling constraint. This finding suggests AI training can be treated as a systematic discipline rather than an unpredictable process.