A/B test models in production

Every AI team eventually hits the same wall: the new model looks great on benchmarks, but nobody knows if it actually performs better for real users. Together AI's answer is a feature that lets you run A/B experiments directly on a live endpoint, splitting traffic between one control deployment and up to 20 variant deployments, each assigned a fixed percentage that must add up to 100.

The pitch here is really about where the complexity lives. Most teams currently bolt this together themselves, with a feature flag or a hash on user ID in client code, two hardcoded model endpoints, and inevitably a spreadsheet somewhere explaining what "group A" even means. That works until it doesn't: the routing logic ships baked into the application, cohort splits drift as clients cache decisions, and the branching code often outlives the experiment because nobody's confident it's safe to delete. Together AI's approach keeps that logic at the endpoint level instead, so the client never even sees a difference, same endpoint, same API, same keys.

Mechanically, the control deployment is the only entry point in the traffic split, and once a request lands there, the platform re-samples it among the experiment's arms according to their configured percentages. Variant deployments have to carry zero weight in the base split, similar to how a shadow deployment is set up, so their entire traffic share comes purely from the experiment. Ramping is handled by simply resending the full member list with an updated percentage split, protected by an etag so a teammate's concurrent change gets rejected rather than silently overwritten. In one live run, Together AI walked an experiment from 95/5 up to 80/20 and then 50/50, and the observed traffic shares tracked the configured ones closely, landing at 95.3/4.7, 79.2/20.8 and 50.2/49.8 across roughly 1,300-plus requests each stage.

Measuring quality is left deliberately to the user. The platform doesn't try to guess what "better" means, it just makes sure every response is tagged with the deployment that served it, so product teams can join that against their own metrics, like ratings or task completions, using the deployment ID as the key. Ending an experiment is a two-step move: promote the winner via a blue-green rollout, then delete the experiment, and traffic falls back to the endpoint's base split automatically. If the variant loses instead, deleting the experiment alone sends 100% of traffic back to the control, no cleanup code required. In the recorded run, after deletion, 360 consecutive requests all landed back on the control with nothing left over to unwind.

A few operational details stand out too. Deployments are monitored and autoscaled independently, so a variant that starts misbehaving only affects its own cohort rather than dragging down the control. Cohort assignment is sticky if a request carries a stable key like a user field, letting the same user stay in one arm across a session, though without a key, assignment is effectively random per request, which is why Together AI's own test data lined up so tightly with the configured splits.