Introducing Amazon SageMaker HyperPod Inference Gateway
Amazon Web Services Vinay Arora ● Covered by 2 sources
AWS launched a Kubernetes add-on that routes LLM requests using live GPU signals. It claims up to 82% lower first-token latency and less wasted GPU capacity.
Based on reporting by Amazon Web Services, Vinay Arora — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is pitching a new fix for one of the ugliest parts of running large language models at scale: waste. The company’s SageMaker HyperPod Inference Gateway is now available as a Kubernetes-native add-on for EKS, and it is designed to stop sending traffic to pods that are already busy, memory-stressed, or holding the wrong adapter in GPU memory.
The basic complaint is familiar. Standard load balancers know nothing about KV cache pressure, queue depth, LoRA residency, or whether a pod is halfway through a long-context response. So round-robin keeps feeding traffic to the wrong places, latency climbs, and teams over-provision to keep service alive. AWS says that can push first-token latency past 4 seconds during bursts.
The gateway tries to fix that with two layers. The first sits inside each HyperPod cluster as the amazon-sagemaker-hyperpod-inference add-on. It uses the Gateway API Inference Extension, an Envoy proxy, a body-based router that reads the model field in OpenAI-compatible requests, and an endpoint picker that scores backends using live Prometheus metrics. Those scores factor in KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit rate, and running requests. A second layer, the Global Inference Router, is planned for cross-cluster and cross-region coordination.
AWS says setup takes a single add-on install and one InferenceGatewayConfig custom resource. No sidecars. No service mesh. No app code changes. Existing clients keep talking standard HTTP to an OpenAI-compatible endpoint, and the gateway can route multiple models from the same cluster based on the request body. It also handles LoRA adapters by preferring pods that already have the right adapter loaded.
The pitch comes with some sharp numbers. In one example, AWS says a chatbot user waiting 4.4 seconds for the first token can get it in under 800 milliseconds. Across benchmarks on four models from 8B to 235B parameters, the company reports big gains versus Kubernetes round-robin routing in mixed GPU, bursty, and shared-prefix workloads, including reductions in P95 and P99 TTFT and, in some cases, higher throughput. On a uniform fleet with steady traffic, though, the gateway was only comparable to round-robin. That is probably the most honest sentence in the whole announcement.
My take — AI-written commentary, not fact-checked reporting
This is the kind of infrastructure work that actually matters, which is rare enough to deserve a nod. Everyone loves to shout about bigger models; fewer people want to talk about the routing mess underneath, where naive balancing turns expensive GPUs into decorative furniture. AWS is also making the right bet on boring openness here: Kubernetes-native, OpenAI-compatible, no app rewrite. That’s how adoption happens, not through another shiny control plane brochure.
Read more about this at: Amazon Web Services