Reduce inference cold starts on Amazon SageMaker HyperPod with model caching

Amazon Web Services · 1 month ago · 6

Amazon SageMaker HyperPod inference experienced long pod cold-start times because each new pod had to pull the vLLM/LMI container image from Amazon ECR and download model weights from the configured remote source before it could serve requests. For a 600+ GB model such as DeepSeek-R1, the article says this can take 30 minutes or more for a single request and repeats on every scale-out event. Amazon launched model caching that pre-loads weights to node NVMe (about 7 GB/s) and can pre-pull images so pods typically start serving in seconds instead of tens of minutes, with fallbacks to normal downloading when cache-ready capacity isn’t available.