Mixture of Experts Explained

Hugging Face Blog · 2 years ago

Mixture of Experts replaces dense transformer feed-forward layers with sparse layers containing multiple expert networks selected by a gating router, enabling faster pretraining and inference compared to dense models of equivalent quality. Mixtral 8x7B requires 47 billion parameters in VRAM despite having 56 billion total parameters, because only feed-forward layers are treated as experts while other parameters are shared across the model. MoEs trade memory overhead and fine-tuning difficulties for significant compute savings during pretraining, with inference speed comparable to a 12-billion-parameter dense model when using two active experts per token.