Serving MiniMax-M3 for efficient inference: Unlocking 1M-Token Context and Multimodality Without Regrets

Together AI just landed the kind of deal every inference startup wants: official cloud partner status for MiniMax M3, a model ambitious enough to make most serving stacks sweat. Once M3 ships as open weights in the coming days, Together will run it both for MiniMax's own production traffic and as a developer endpoint. That's a meaningful vote of confidence, because M3 isn't a simple decoder-only model you can drop onto existing infrastructure and call it a day.

The hard part is architectural. M3 introduces MiniMax Sparse Attention, which caps how many tokens each query can attend to instead of letting attention cost balloon quadratically with context length. That's what makes a 1-million-token window practical rather than theoretical. MiniMax claims MSA alone delivers more than a 9x speedup during prefill and over 15x during decoding compared to the prior M2.7 generation. Add native multimodal support — real image and video understanding baked into the model rather than bolted on — and you've got a system that breaks a lot of assumptions baked into standard GPU serving kernels.

So Together's engineers rewrote several of those assumptions from scratch. They built a KV-block-major attention kernel that flips the usual query-outer, key-inner loop order, cutting redundant memory movement between HBM and SRAM during prefill. They found a way to bolt MSA onto existing paged-attention kernels by flattening the KV-group dimension into the batch dimension and exploiting tensor striding, avoiding a full kernel rewrite and picking up a 5% decode throughput gain in the process. And because MSA shifts cost toward a top-k scoring step that runs on every single decode token, they built a dedicated indexing kernel using an AB-swapped HMMA layout to keep that scan from becoming the new bottleneck.

There's also a less glamorous but equally consequential piece: a Rust-based gateway called SMG that handles all image and video preprocessing — decoding, frame sampling, resizing, patchifying — before a request ever touches the GPU. That keeps expensive CPU work off the accelerator entirely, which matters a lot once video frames start showing up in agentic workflows alongside 1M tokens of text.

Put together, Together AI says these changes produced throughput gains of 81% to 125% across different concurrency levels under realistic agentic traffic patterns. That's not a marginal tuning win; it's the difference between a frontier model being economically viable to serve at scale or staying a research curiosity. The team is already eyeing further kernel fusion and disaggregating KV-cache loading from index loading, which suggests M3's sparse attention design still has headroom nobody has fully extracted yet.