Mixture of Experts (MoEs) in Transformers

Hugging Face Blog · 4 months ago

Mixture of Experts models replace dense feed-forward layers in transformers with multiple expert sub-networks, where a router selects only a small subset of experts to process each token, enabling models with many more total parameters to run with fewer active parameters per inference step. The transformers library improved MoE model loading speed from 66 seconds to 10 seconds on a 110B-parameter model by implementing dynamic weight conversion that transforms checkpoint tensors into GPU-optimized packed layouts without unnecessary materialization. The improvements enable efficient quantization, pluggable expert execution backends, and distributed expert parallelism across multiple devices, making sparse architectures practical for both training and inference at scale.