Foundational research powering efficient inference at scale

Everyone still talks about training runs like they're the main event, but Together AI's latest writeup makes a blunt point: for teams actually shipping AI products, inference is where the money goes. Not once, not occasionally, but on every single user message, every agent step, every API call. The company cites estimates putting inference at 80 to 90 percent of a production AI system's total lifetime cost, which reframes the whole conversation. Training is a one-time (or periodic) expense you can budget for. Inference scales with success, which is either great news or a slow-motion margin problem depending on how well your stack handles it.

Ce Zhang, Together's CTO, used a GTC talk to walk through why this is genuinely hard, not just expensive. Latency isn't cosmetic anymore — chain five model calls together at 200 milliseconds each and you've burned a full second before a user sees anything, which is the difference between a product feeling instant and feeling broken. Throughput is the other lever, and it's a brutal one for AI-native companies: where traditional SaaS runs 80-90 percent gross margins, AI companies are commonly stuck at 50-60 percent, with inference alone chewing through roughly 23 percent of revenue at scaling-stage startups. And on top of that, the ground keeps moving — new model architectures and quantization tricks mean an inference stack tuned for today's models can be stale within months.

Together's pitch is that it treats inference as a compounding research problem rather than a one-off tuning job. The company points to its own lineage of published work — FlashAttention, now on its fourth iteration, plus ThunderKittens and an open-source speculative decoding system called Aurora — landing in production for customers within weeks of release. Aurora is the more interesting piece here: standard speculative decoding uses a small draft model to guess tokens that a bigger model verifies in parallel, typically buying 1.5 to 3x speedups on predictable tasks like code completion. Aurora instead learns continuously from live traffic using reinforcement learning, adapting as usage patterns shift instead of going stale like a static draft model would. Together says it delivers around 1.25x faster inference on top of already-optimized baselines, without pausing serving to retrain.

The rest of the stack is less flashy but arguably just as load-bearing: custom parallelism across 72-GPU Blackwell meshes, NVFP4 quantization, and real-time batching decisions about which requests to group and how to route them based on context length. Together name-drops Cursor as a case where this mattered in practice — millions of active developers hitting strict latency SLAs under unpredictable, high-concurrency load isn't a problem you solve with a bigger GPU order, it's a scheduling and engineering problem.

The bigger irony the piece surfaces, via Stanford's 2025 AI Index, is that inference costs for GPT-3.5-level performance fell more than 280-fold between late 2022 and late 2024, and yet total inference spend keeps climbing anyway. Cheaper tokens didn't shrink the infrastructure problem, they just convinced everyone to use ten times more tokens. That's the actual argument for why inference optimization is a durable moat rather than a commodity checkbox: run inference twice as efficiently and you're not just saving money on the same workload, you're unlocking use cases that weren't economically viable a quarter ago.