Run multi-node inference workloads using Ray LLM Deep Learning Containers
Amazon Web Services Ananth Raghavendra
AWS shows how to split one LLM across two GPU nodes on EKS. Prefill and decode run separately, so long prompts stop clogging token generation.
Based on reporting by Amazon Web Services, Ananth Raghavendra — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is pushing multi-node inference a step beyond the usual single-node demo. In a new post, it shows one large language model running across two GPU workers on Amazon EKS, with one node handling prefill and the other handling decode behind a single OpenAI-compatible endpoint.
That split matters because the two phases behave very differently. Prefill chews through the whole prompt and builds the KV cache. Decode then reads that cache and emits tokens one by one. Put both on the same GPU and they fight over the same resources; separate them and the compute-heavy part stops tripping up the latency-sensitive part.
The stack AWS uses is the Ray LLM Deep Learning Container, which bundles Ray Serve LLM, vLLM, NVIDIA Inference Transfer Library, and the EFA networking pieces. The application itself is described entirely in YAML through build_pd_openai_app. There’s no custom handler class, no model archiver, and no hand-built image to maintain.
The layout is pretty specific. A CPU-only Ray head runs on an m7i.xlarge system node, while two g6.8xlarge workers carry the model. One worker serves prefill, the other decode. The GPU node group is pinned to a single Availability Zone because EFA cannot cross AZs, and the KV cache moves between workers over NIXL using libfabric.
AWS also calls out some practical tuning. The example uses qwen3.5-9b with max_model_len set to 4096, bfloat16, and gpu_memory_utilization at 0.90. It says adding kv_connector_extra_config with LIBFABRIC is critical, because otherwise NIXL falls back to UCX and runs at TCP speed even on EFA. With the cache settings in place, the post says model weight loading drops from around 145 seconds to about 2 seconds on restart.
My take — AI-written commentary, not fact-checked reporting
This is the kind of cloud plumbing that matters more than another shiny demo. AWS is basically saying: stop hand-stitching GPU, CUDA, and networking pieces every time you want distributed inference to work. Sensible, boring, and exactly the sort of boring that saves teams from setting money on fire.
Read more about this at: Amazon Web Services