Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters
Qwen
Alibaba's Qwen team released Qwen1.5-MoE-A2.7B, a mixture-of-experts language model with 2.7 billion activated parameters that matches the performance of 7-billion-parameter models like Mistral 7B and Qwen 1.5-7B on benchmarks including MMLU, GSM8K, and HumanEval. The model reduces training costs by 75% and achieves 1.74 times faster inference speed compared to the standard Qwen 1.5-7B, while using only one-third of the non-embedding parameters. The efficiency gains enable practitioners to deploy comparable model quality with substantially lower computational requirements for both training and inference.
Why it matters
GITHUB HUGGING FACE MODELSCOPE DEMO DISCORD Introduction Since the surge in interest sparked by Mixtral, research on mixture-of-expert (MoE) models has gained significant momentum. Both researchers and practitioners are keenly interested in understanding how to effectively train such models and assessing their efficiency and effectiveness. Today, we introduce Qwen1.5-MoE-A2.7B, a small MoE model with only 2.7 billion activated parameters yet matching the performance of state-of-the-art 7B models like Mistral 7B and Qwen1.