Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components
Amazon Web Services Rielah De Jesus
Salesforce used a new SageMaker setting to spread AI model copies across two AWS zones. That turned cheaper inference into something that could also pass a stricter uptime rule.
Based on reporting by Amazon Web Services, Rielah De Jesus — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Salesforce wanted Agentforce, its AI system for agents, to stay available across multiple AWS Availability Zones. SageMaker Inference Components had already helped cut infrastructure costs by letting several models share GPUs, with Salesforce seeing an 8x reduction. But the default placement logic did not promise the kind of zone-by-zone resilience the company needed for production.
The problem was simple enough, and annoying in the way infrastructure problems always are. SageMaker’s default placement works one deployment at a time. That keeps things efficient, but it can also pile a model’s copies into the same zone or even the same instance. If that instance fails, or if an entire Availability Zone goes dark, the model can go with it. Salesforce needed every production model to span two AZs, not just sit behind a multi-AZ endpoint and hope for the best.
AWS responded with SchedulingConfig in the CreateInferenceComponent API. The key pieces are AvailabilityZoneBalance, which tries to spread copies across zones, and PlacementStrategy, which lets customers choose SPREAD or BINPACK within a zone. Salesforce used SPREAD for fault isolation, then set MaxImbalance to 1 for a four-copy model so the system would keep the zone split close to even. For a lighter model with just two copies, MaxImbalance 0 gives a strict one-copy-per-AZ layout.
That placement control matters during scaling too. When Salesforce scales out, SageMaker places new copies to preserve the zone split. When it scales in, copies are removed symmetrically across zones. And for longer-term cleanup, the endpoint can use a CONSOLIDATION ScaleInPolicy so idle instances get released without breaking the balance rules. One warning from the post is blunt: never drop a high-availability model to CopyCount 1, because that instantly destroys two-AZ compliance.
The broader lesson is that endpoint-level redundancy is not enough if the copies themselves are clustered badly. Salesforce also pre-provisions GPU capacity with On-Demand Capacity Reservations, then watches SageMaker AI Insights metrics such as AZ skew, copy count per AZ, rebalancing events, and insufficient capacity errors. The result is a setup that keeps the cost savings from shared GPUs while meeting a stricter availability bar.
My take — AI-written commentary, not fact-checked reporting
This is the unglamorous truth of enterprise AI: the hard part isn’t getting a model to run, it’s getting it to behave like production software. Salesforce didn’t need more hype; it needed placement controls, capacity planning, and a way to stop one bad zone from taking out the whole show. That’s the real story here, and it’s why the “just use the cloud” crowd keeps running into the same wall.
Read more about this at: Amazon Web Services