Global-batch load balance almost free lunch to improve your MoE LLM training
Qwen
Researchers propose global-batch load balancing for training Mixture-of-Experts large language models, replacing the standard micro-batch approach to better encourage expert specialization. Testing on models from 3.4B to 43B parameters showed consistent performance improvements across all configurations, with perplexity decreasing rapidly as balance batch size increased from 2 to 128. The method requires minimal computational overhead since expert frequency synchronization is nearly free, and adding micro-batch loss constraints maintains inference speed at 1.59 seconds per update step while preserving gains.
Why it matters
GITHUB HUGGING FACE MODELSCOPE DISCORD Background The Mixture-of-Experts (MoEs) architecture has become a popular model-parameter-scale-up technique. Typically, one MoE layer consists of a router (often parameterized as one single Linear layer) and a group of experts (for transformer-based models, each expert is one feedforward layer). Given an input, only a subset of experts will be activated, and then their outputs will be aggregated based on the scores the router assigned.
