Welcome Mixtral - a SOTA Mixture of Experts on Hugging Face
Hugging Face ● Covered by 4 sources
Mistral dropped Mixtral 8x7B, a new open-weight AI model, and Hugging Face rushed to plug it into their whole toolkit. It beats GPT-3.5 on most tests and matches Llama 2 70B's rivals while running like a much smaller model.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Mistral's latest release isn't just another open model dump. Mixtral 8x7B uses a Mixture-of-Experts design, meaning it's technically eight expert sub-networks stitched together, though only two get activated per token. The result: it runs at roughly the speed of a 12-billion-parameter model while carrying the knowledge of something far bigger. Hugging Face wasted no time wiring it into Transformers, Inference Endpoints, and Text Generation Inference, plus shipping a fine-tuning recipe using TRL.
The naming is a bit of a trap, and Hugging Face spent real effort correcting the record. People assumed "8x7B" meant 56 billion parameters, or eight separate 7B models glued together. Neither is true. Only the feed-forward blocks get replicated eight times; everything else stays shared, landing the real parameter count around 45 billion. A cleaner name, Hugging Face admits, would've been something like Mixtral-45-8e.
On benchmarks, the numbers are genuinely striking. The base model tops Llama 2 70B on the Open LLM Leaderboard, and the Instruct-tuned version is the first open-access model to hit GPT-3.5-level scores on MT-Bench. It handles 32k tokens of context, speaks five languages, and scores 40.2% on HumanEval for coding tasks. All under an Apache 2.0 license, so commercial use isn't an issue.
Running it, though, is where things get expensive fast. Full float16 precision demands over 90GB of VRAM — Hugging Face itself flags this with a stunned emoji in their own post. Drop to 4-bit quantization via bitsandbytes or GPTQ and you can squeeze it into 23-30GB, putting a single A100 or A6000 within reach. TheBloke's GPTQ version loses barely anything in quality, with perplexity climbing from 4.25 to 4.40. Fine-tuning on a single A100 using QLoRA takes about 48 hours, and Hugging Face notes you should skip targeting the MLP layers since they're sparse and don't play nice with PEFT.
What's missing from Mistral's release is notable too. There's no disclosure on pretraining dataset size, composition, or how it was cleaned, and nothing about the fine-tuning data or hyperparameters behind the instruct version's SFT and DPO stages. Open weights, opaque process — a pattern that's becoming the norm rather than the exception in this corner of AI.
My take — AI-written commentary, not fact-checked reporting
Mixtral is proof that Europe can ship a frontier-adjacent model without pretending to be OpenAI about it — Apache 2.0, no waitlist, no mystery API. But the total silence on training data is the tell: "open" model releases keep meaning open weights, not open science, and nobody in this industry seems eager to change that math.
Read more about this at: Hugging Face