Fault tolerant distributed training on Amazon EKS using NVRx
Amazon Web Services Aravind Neelakantan
AWS shows how NVRx keeps long EKS training runs moving after GPU faults or hangs. It also cuts checkpoint stalls that can eat up to 40% of wall time.
Based on reporting by Amazon Web Services, Aravind Neelakantan — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is pitching NVRx as a way to make big PyTorch FSDP jobs on Amazon EKS less fragile and less wasteful. The target is the ugly reality of multi-hour, multi-day training runs across dozens of nodes: one worker hiccups, NCCL starts timing out, healthy pods crash out of sync, and the cluster keeps burning GPU hours without making progress.
The post’s other problem is checkpointing. Synchronous saves force every rank to wait on I/O, and on the cluster sizes AWS used, that idle time reached as much as 40% of total wall time. NVRx attacks both sides of that pain by dropping into an existing script as a pip-installable layer, with no custom kernels, no PyTorch fork, and no recompile.
The first piece is async checkpointing. With TorchAsyncCheckpoint, the save path hands state to a background process and returns right away, while each rank writes its own shard with FSDP LOCAL_STATE_DICT. That avoids the all-gather and rank-0 bottleneck that come with the traditional approach. The second piece is in-process restart, which keeps the Python process alive when there’s an unhandled exception or an NCCL hang. NVRx aborts the active process group, checks rank health, re-rendezvouses the survivors, and re-enters training from the latest checkpoint.
For harder failures, AWS points to ft_launcher, the NVRx in-job restart launcher. That covers SIGKILL, OOM kills, and OS-level hangs by watching rank heartbeats and respawning workers in the same job when something stalls or dies. The post frames the three layers as separate tools for separate fault classes: in-process for soft faults, ft_launcher for hard faults, and the cluster orchestrator for node loss.
The reference setup is a managed EKS cluster with self-managed p5.48xlarge nodes, each with 8 NVIDIA H100 80 GB GPUs and 32 EFA interfaces. Checkpoints land on Amazon FSx for Lustre, mounted into every pod, and AWS says keeping that filesystem in the same Availability Zone as the GPU nodes helps recovery because checkpoint loading, not the restart mechanism itself, dominates recovery time at scale. The post also says the benchmark results were run on H100 GPUs from 2-node to 8-node scale, with code available to reproduce.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of boring: fewer heroic restarts, more plumbing that actually works. The industry keeps pretending training reliability is a research problem when half the pain is just bad failure handling and synchronous I/O. Closed stacks love to sell resilience; open tooling like this is what makes it real.
Read more about this at: Amazon Web Services