Together AI delivers fastest inference for the top open-source models
Together AI
Together AI now runs top open-source models like DeepSeek and Kimi-K2 up to 2x faster than any other GPU provider, per independent benchmarks. The gains come from Blackwell-tuned kernels and custom speculative decoding, not just bigger chips.
Based on reporting by Together AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Speed has quietly become the real battleground in open-source AI hosting, and Together AI just posted numbers that are hard to ignore. According to independent benchmarks from Artificial Analysis, the company now leads output-speed rankings among GPU-based providers for a long list of heavyweight models: GPT-OSS-20B and 120B, Qwen3-235B, Qwen3-Coder-480B, Kimi-K2, DeepSeek-R1, and DeepSeek-V3.1. Kimi-K2-0905 in particular is running 65% faster than the next-best provider, and Qwen3-235B is up over 2.75x.
The gains didn't come from one clever trick. Together rebuilt its inference stack from the ground up to match NVIDIA's Blackwell architecture, including the GB200 NVL72 chips, tuning kernel design, memory layout, and scheduling to work as one system rather than a pile of separate optimizations. Custom kernels — including a version of FlashAttention-4 and fused mixture-of-experts kernels — squeeze more throughput out of the same silicon by cutting overhead in how attention and routing computations move through memory.
Quantization is the second lever, and it's the one that usually forces a tradeoff between speed and accuracy. Together says its calibration pipeline compresses model weights down to FP8 and FP4 formats while staying close to lossless, using block-wise scaling and selective mixed precision on the parts of a model most sensitive to compression. That's a meaningful claim, since aggressive quantization at this scale often degrades output quality in ways that only show up on harder tasks.
The most interesting piece, though, is the speculative decoding work. Instead of generating tokens one at a time with the full model, Together trains smaller
My take — AI-written commentary, not fact-checked reporting
The benchmarks are real and the Blackwell optimization work sounds genuinely hard, but let's not pretend this is charity — Together is selling inference, and speed is the only lever left once everyone has access to the same open weights. Still, cheaper, faster serving of DeepSeek and Kimi-K2 is exactly the kind of infrastructure that keeps open models competitive with closed ones on cost, which matters more than any single benchmark chart.
Read more about this at: Together AI