Together AI Delivers Top Speeds for DeepSeek-R1-0528 Inference on NVIDIA Blackwell
Together AI
Together AI now runs DeepSeek-R1 on Nvidia's newest Blackwell chips, hitting claimed record inference speeds. It's the fastest serverless setup yet for a top open-source reasoning model, without dumbing down accuracy.
Based on reporting by Together AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Together AI says it's cracked a new speed ceiling for DeepSeek-R1-0528, the open-source reasoning model that's become a genuine rival to closed frontier systems. Running on Nvidia's HGX B200 Blackwell GPUs, the company's custom inference stack reportedly hits peak throughput of 334 tokens per second, about 32 tokens faster than the same hardware running a standard open-source inference engine, and 2.3x to 2.8x quicker than the prior-generation H200 chips. As of July 17, 2025, Together claims that's the fastest serverless DeepSeek-R1 deployment anywhere.
The gains come from stacking several optimizations on top of each other rather than relying on raw silicon. Together built its own GPU kernels using the ThunderKittens framework, matching Nvidia's own performance benchmarks within two weeks of getting Blackwell hardware in hand. It also developed a speculative decoding system called Turbo Speculator, which predicts several tokens ahead using a smaller model before the full model verifies them in parallel — and Together says its version beats not just open-source alternatives but even DeepSeek's own 14-billion-parameter MTP module, especially as the lookahead window grows.
Quantization is the other lever here. Together's team developed what it calls a lossless approach to NVFP4, a 4-bit numerical format Blackwell supports natively, compressing model weights 3.6x versus standard BF16 while claiming almost no accuracy hit, even in attention layers where compression usually causes trouble. The company backs this up with LiveCodeBench testing across 267 coding tasks, arguing its quantized deployment tracks near point-for-point with uncompressed runs, though it flags that a
My take — AI-written commentary, not fact-checked reporting
The gains here are real, but let's be clear about what's happening: this is Together renting you speed on Nvidia's chips using their own proprietary software glue, not some open breakthrough anyone can replicate. That's the uncomfortable irony of the open-weights era — the model is open, but the fastest way to run it well is increasingly locked behind a handful of infra companies with kernel teams and Blackwell access. I'd rather see more of this optimization work actually open-sourced, the way the ThunderKittens kernels partly were, than treated as a walled-garden selling point."} (Note: formatting issue above)
Read more about this at: Together AI