Benchmarking inference at scale: coding agents

Together AI · 2 months ago

Together released a benchmark measuring inference performance on production coding agent workloads, where Together Inference Engine delivers 31% higher throughput than TensorRT-LLM and maintains 2× better time-to-first-token at saturation of 2.5M tokens per minute. The company optimized performance through ThunderMLA (a fused kernel for Multi-head Latent Attention), custom kernels, and full-stack profiling rather than relying on single-user benchmarks. For typical coding requests of 80k-100k input tokens, Kimi K2.7 on Together costs $0.029 per request compared to $0.097 for Claude Opus, saving an engineering team of 150 people approximately $421k annually.