Global-batch load balance almost free lunch to improve your MoE LLM training

Qwen · 1 year ago

Researchers propose global-batch load balancing for training Mixture-of-Experts large language models, replacing the standard micro-batch approach to better encourage expert specialization. Testing on models from 3.4B to 43B parameters showed consistent performance improvements across all configurations, with perplexity decreasing rapidly as balance batch size increased from 2 to 128. The method requires minimal computational overhead since expert frequency synchronization is nearly free, and adding micro-batch loss constraints maintains inference speed at 1.59 seconds per update step while preserving gains.