18 summarised stories about Mixture-of-Experts, each linking back to the original source. Browse all topics →
Mistral AI 2 years ago
Mistral AI released Mixtral 8x7B, a sparse mixture-of-experts language model with open weights under Apache 2.0 license. The model has 46.7 billion total parameters but uses only 12.9 billion per token, matching or outperforming GPT-3.5 and Llama 2 70B on most benchmarks while running 6 times faster. Developers can now deploy the model using open-source tools like vLLM and access it through Mistral's API platform.
Hugging Face Blog 2 years ago
Mixture of Experts replaces dense transformer feed-forward layers with sparse layers containing multiple expert networks selected by a gating router, enabling faster pretraining and inference compared to dense models of equivalent quality. Mixtral 8x7B requires 47 billion parameters in VRAM despite having 56 billion total parameters, because only feed-forward layers are treated as experts while other parameters are shared across the model. MoEs trade memory overhead and fine-tuning difficulties for significant compute savings during pretraining, with inference speed comparable to a 12-billion-parameter dense model when using two active experts per token.
Hugging Face Blog 2 years ago
Mistral released Mixtral 8x7B, a mixture-of-experts language model that outperforms GPT-3.5 on most benchmarks and is now integrated across Hugging Face's ecosystem including Transformers, Inference Endpoints, and Text Generation Inference. The model contains 45 billion effective parameters across 8 specialized experts, decodes at the speed of a 12-billion parameter model, and achieves 40.2% accuracy on HumanEval coding tasks. Users can now run inference, fine-tune on single GPUs, and deploy Mixtral through multiple Hugging Face tools with support for quantization to reduce memory requirements from 90GB to 23GB.
