Reduce inference cold starts on Amazon SageMaker HyperPod with model caching
Amazon Web Services Kareem Syed-Mohammed
AWS added model caching to SageMaker HyperPod. It can cut LLM startup from tens of minutes to seconds by keeping weights and images on local NVMe.
Based on reporting by Amazon Web Services, Kareem Syed-Mohammed — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is trying to kill one of the ugliest parts of running large models on SageMaker HyperPod: the wait. When a pod comes up today, it can spend a long time pulling an inference image from Amazon ECR and then even longer fetching model weights from Amazon S3, Amazon FSx for Lustre, or HuggingFace Hub. For smaller models, that’s a few minutes. For something like DeepSeek-R1 at 600+ GB, the delay stretches past 30 minutes before the first request lands.
The new feature, model caching for Amazon SageMaker Inference on HyperPod, moves both of those downloads closer to the metal. Weights cache preloads model files onto local NVMe on each node, and the operator waits until the target nodes are marked cache-ready before it creates the inference deployment. Once the pod starts, it reads from local storage at about 7 GB/s. Image cache does a similar trick for the serving container itself, pre-pulling the image so pods don’t sit around waiting for ECR.
AWS says the two caches can be used together or separately. Weights cache is the more opinionated one: it’s tied to the deployment lifecycle and can keep the cached files around across restarts on the same node. Image cache is looser. The deployment is created right away, and the DaemonSet keeps pulling the image onto target nodes in the background. If a pod lands before the pull finishes, it falls back to the normal ECR download.
That fallback matters. AWS is using preferred scheduling, not required scheduling, so caching speeds things up without blocking startup. A pod can still launch on a cold node; it just pays the old network penalty. The same applies during scale-out. If traffic jumps faster than the cache has spread, some pods still go through the full download path.
The pitch is straightforward: fewer cold starts, faster scale-out, and less dependence on network throughput to the storage backend. AWS says benchmarks across models from 57 GB to 145 GB showed around 60 percent faster scale-out with weights caching, while image caching can remove over two minutes of image-pull time and cut that step by up to 97 percent. The feature is generally available now in all regions where HyperPod is available.
My take — AI-written commentary, not fact-checked reporting
This is the kind of boring infrastructure feature that actually matters. Everyone loves talking about model quality; nobody enjoys watching a pod spend half an hour downloading itself into existence. AWS is basically admitting that inference ops is still mostly a glorified waiting game, and local caching is the sensible fix.
Read more about this at: Amazon Web Services