Mixtral of experts

Mistral AI just handed the open-source world a genuinely strong toy. Mixtral 8x7B isn't a bigger version of the same old transformer recipe — it's a sparse mixture-of-experts model, meaning it has 46.7 billion parameters total but only activates 12.9 billion of them for any given token. Eight expert groups sit inside each layer, and a router picks two to handle each piece of text. The upshot: you get the quality benefits of a much larger model while paying the compute and latency costs of one roughly the size of a 13B model.

The benchmark numbers back up the architecture bet. Mixtral outperforms Llama 2 70B on most standard tests while running six times faster, and it goes toe-to-toe with GPT-3.5 on many of them too. That's a big deal for a model released under the permissive Apache 2.0 license — no usage restrictions, no asking permission. Mistral is positioning it as the best cost-to-performance open model available right now, and the benchmark charts they published make a decent case for that claim.

Beyond raw scores, Mixtral handles a 32,000-token context window, speaks five languages fluently (English, French, German, Spanish, Italian), and writes competent code. There's also an instruction-tuned variant, Mixtral 8x7B Instruct, refined through supervised fine-tuning and direct preference optimization. It scores 8.3 on MT-Bench, which Mistral says makes it the top open-source model on that leaderboard, sitting comfortably near GPT-3.5 territory. On bias testing, Mixtral reportedly shows fewer problems than Llama 2 on the BBQ benchmark and skews more positive on BOLD sentiment checks — though base models will follow any instruction you give them unless you explicitly prompt for moderation.

Mistral isn't just throwing weights over the fence and walking away. The company contributed code to the vLLM project, integrating Megablocks CUDA kernels so the community can actually run this thing efficiently without proprietary infrastructure. Pair that with Skypilot for cloud deployment and you've got a fairly complete open pipeline. Mixtral is also live behind Mistral's own mistral-small endpoint in beta, for anyone who'd rather not manage their own GPUs.