Fault tolerant distributed training on Amazon EKS using NVRx
Amazon Web Services 4 weeks ago 39
NVIDIA Resiliency Extension (NVRx) was integrated with PyTorch FSDP on Amazon EKS so distributed training can recover from worker faults and reduce wasted time from checkpoint stalls. The post reports synchronous checkpointing consumed up to 40% of total wall time on cluster sizes used for its benchmarks on H100 GPUs. With NVRx, checkpointing is made async and training can restart in-process for soft faults or via ft_launcher for hard faults, cutting downtime while preserving progress.