Configuring Dedicated Model Inference
Together AI ● Covered by 2 sources
Together AI explained how its dedicated inference platform routes traffic based on actual server capacity, not fixed percentages. That design means autoscaling and traffic splitting happen automatically together, without extra config work.
Based on reporting by Together AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Together AI just published a deep dive into the plumbing behind its Dedicated Model Inference product, and it's more interesting than the average infrastructure post because it explains a design decision that most platforms get wrong: how traffic gets divided between multiple copies of a model running behind one endpoint.
The system splits into three pieces. An endpoint is the stable name your app calls, something like myproject/my-endpoint, and it never changes even if you swap hardware underneath it. A deployment binds a specific model version to a specific config and runs some number of replicas. And a config is an immutable recipe, GPU type, count, parallelism, whether it's tuned for latency or throughput, each one stamped with an ID like cr_8x7f2. Because configs can't be edited, only replaced by a new revision, you always know exactly what a deployment was tested against, and you always have a rollback target.
The clever part is the traffic split. Instead of assigning fixed percentages to each deployment, Together assigns a weight per replica, and the router multiplies weight by the number of ready replicas to get effective capacity. Two deployments with equal weight but different replica counts get traffic proportional to those replica counts, not a 50/50 split. Scale one deployment from one replica to three and it automatically absorbs three times the share, no manual rebalancing required. Replicas that are still cold-starting or unhealthy contribute zero capacity, so the router simply stops sending them anything until they're ready again.
Together ran this live on two single-H100 deployments to prove it out. With one replica each, a 50/50 split showed up as roughly 47.6/52.4 across 599 requests, close enough to the model's own noise. Scale one deployment to two replicas and the expected 66.7/33.3 split showed up as 69.4/30.6. They also benchmarked two serving profiles head to head: one profile handled concurrency of 16 at 1,464 tokens per second with barely rising latency, while the other profile was actually faster at low concurrency but collapsed hard once load increased, throughput cut in half and requests starting to stall. The lesson buried in that chart is blunt: if you only load-test at low concurrency, you'll pick the wrong config for production and not find out until it's already breaking.
All the fancier features, canary rollouts, A/B tests, shadow traffic, are really just this same mechanism reused. An A/B test is deployments with cohort labels attached. A shadow experiment is a deployment sitting at weight zero that quietly receives mirrored requests. Stopping a deployment is just capping its replica count at zero. Together built one primitive and let everything else fall out of it, which is a cleaner story than most inference platforms tell.
My take — AI-written commentary, not fact-checked reporting
This is the kind of infra writeup I actually trust, because it shows the seams instead of hiding them, including the exact gotcha where a fresh deployment silently gets zero traffic until someone remembers to add it to the split. Capacity-aware routing is objectively the right call over fixed percentages, and it's a little embarrassing that this isn't table stakes everywhere by now. My only gripe is that Together, like every other hosted provider, is still asking you to trust a black box GPU fleet you can't inspect — fine for prototyping, but anyone serious about sovereignty or cost control should be asking why this same weighted-capacity logic can't just live in an open-source router sitting in front of your own hardware.
Read more about this at: Together AI