Autoscaling endpoints for LLM inference
Together AI ● Covered by 2 sources
Together AI let you pick which signal triggers autoscaling for LLM inference, not just generic CPU load. Their own tests show GPU utilization and latency metrics missed a real traffic spike that in-flight requests caught immediately.
Autoscaling sounds like a solved problem until you try to apply it to LLM inference, and Together AI's new writeup on their Dedicated Model Inference product makes a pretty convincing case for why the old playbook doesn't transfer. The core issue: GPU utilization and even latency metrics can look calm while your system is actually falling over. In one experiment, they ran the same bursty traffic pattern (a sine wave oscillating between 12 and 48 requests per second, plus two spikes to 80 rps) against a Qwen3.5-9B deployment three separate times, each governed by a different autoscaling metric.
Only one of the three policies actually did anything. Autoscaling on in-flight requests, with a target of 8 per replica, correctly scaled from 1 to 2 to 3 replicas as the load climbed, using 26 replica-minutes total and dropping the error count to 536 out of 40,600 requests. Meanwhile, the policy tuned to TTFT (time to first token) at a 300ms p95 target never scaled at all, because continuous batching absorbed the queueing pressure into overall latency rather than delaying the first token. The gpu_utilization policy, targeting 75%, also never triggered, since the bursty, short requests never pushed raw GPU busy-ness that high. Both of those policies technically served more total requests with fewer errors, but that's misleading: client-side p95 latency was sitting at 3 to 5 seconds the whole time, saturated, while the metrics driving the autoscaler insisted everything was fine.
The deeper problem is architectural. Together AI points out that cold starts for a new replica take one to two minutes at best, and up to two and a half minutes when scaling from one replica to two, once you count GPU placement, weight download, engine load and warmup. A base catalog model like Qwen3.5-9B takes about 86 seconds just to reach ready state; a custom fine-tune with fresh 18GB weights takes 145 seconds. That means any autoscaler reacting to a spike is already too late unless it's watching a leading indicator. In-flight request count rises before latency visibly degrades, which is exactly why Together AI treats it as the safe default, while TTFT and GPU utilization are framed as trailing signals better suited for SLO enforcement or cost optimization once you already understand your workload's shape.
The platform also exposes some blunt operational levers worth knowing. Setting min and max replicas both to zero fully stops a deployment and halts billing, useful for dev endpoints, but there's no scale-to-zero-with-automatic-wake; a stopped deployment returns errors rather than spinning back up on its own. And the asymmetry in tuning windows is deliberate: a short scale-up window costs a few wasted replica-minutes if wrong, but a scale-down window that's too short produces a sawtooth pattern of constant cold starts, which is worse for both cost and latency than just leaving idle capacity running a bit longer.
What's genuinely useful here is Together AI publishing the actual failure mode rather than just the happy path. It's easy to assume utilization-based autoscaling is inherently sensible because it feels efficiency-minded, but this test shows it can quietly let a system saturate while reporting green across the board.
My take
This is a rare vendor post that actually shows you the ways their own defaults could screw you, which earns some trust. My one gripe: everyone building on managed inference platforms should assume utilization and latency metrics lie by default, not treat that as a surprising finding from one company's blog post — it's the same lesson container orchestration learned a decade ago, just with GPUs and worse cold starts this time.
Read more about this at: Together AI
