Cut GPU inference cold start from 8 minutes to less than a minute
The New Stack Sajjan Gundapuneedi
Amazon’s EKS Auto Mode and related platform components were instrumented end-to-end for GPU inference pod startup, revealing six sequential bottlenecks that add up to an 8-minute time-to-first-token response on a 70B-class model. For the 203 GB model, downloading weights from S3 took 423 seconds (about 92% of startup time), partly due to a pattern that left 98% of bandwidth idle. Configuration changes and platform features cut cold-start latency from 8–15 minutes to under 1 minute (with warm-node restarts dropping to under 30 seconds), mainly by fixing S3 weight loading, CUDA kernel compilation caching, and some node/image startup steps.
Why it matters
We instrumented the full path from pod creation to first inference response on a GPU node running a 70B-class model. The post Cut GPU inference cold start from 8 minutes to less than a minute appeared first on The New Stack.