Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2
Amazon Web Services Iman Abbasnejad
Heidi Health, with AWS and NVIDIA, deployed CUDA Multi-Process Service (MPS) and NVIDIA Triton Inference Server on Amazon EC2 to serve its fine-tuned Parakeet TDT ASR model more efficiently. Using MPS reduced the number of GPU instances needed from 16 to 4 while keeping sub-second transcription latency at 92.1 requests per second per GPU. The setup changes production operations by partitioning one L40S GPU into concurrent MPS execution contexts and batching/scheduling requests through Triton, cutting infrastructure cost without increasing latency.
Why it matters
Serving automatic speech recognition (ASR) models at scale is costly when each request uses only a fraction of a GPU. Learn how NVIDIA CUDA Multi-Process Service (MPS) with NVIDIA Triton Inference Server on Amazon EC2 GPU instances cuts GPU infrastructure by 75% while holding sub-second latency at 92.1 requests per second per GPU.