How Deepgram optimized voice AI at scale with AWS Trainium
Amazon Web Services Pierre Lienhart
Deepgram moved its Nova-3 speech model onto AWS Trainium2 for batch jobs. It’s getting up to 70% more audio per dollar on offline transcription.
Based on reporting by Amazon Web Services, Pierre Lienhart — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Deepgram’s voice stack is no side project anymore. Its models cover speech-to-text, text-to-speech, and voice agents, and they now chew through quadrillions of audio tokens a year for more than 200,000 developers and 1,400 organizations.
That scale makes efficiency a business problem, not a nice-to-have. The company wanted to squeeze more audio out of the same infrastructure budget, especially for heavy batch workloads like contact-center analytics and medical transcription, so it worked with AWS to move Nova-3 speech-to-text onto Trainium2.
The port started from standard PyTorch code. Over several months, Deepgram and AWS Forward Deployed Engineering pushed Nova-3 through TorchNeuron, first getting it to run on Trainium2, then compiling the encoder and decoder as separate static-shape graphs, and then replacing some layers with fused kernels from NKILib. Profiling with Neuron Explorer and the NAD tooling pointed to the parts that still needed hand work, including FFN, depthwise conv1d, and conv2d.
The hardware setup is pretty specific: eight model replicas on a single Amazon EC2 trn2.3xlarge instance. On Deepgram’s offline, throughput-oriented batch workloads, that configuration processed 11–70 percent more audio per dollar than the compute options it had been using, based on instance pricing at the time of evaluation.
And the model wasn’t the only thing that changed. Deepgram also built a lightweight Rust runtime so its production serving stack could call Trainium without Python or PyTorch in the path. That runtime uses a work-stealing queue, loads ahead-of-time compiled NEFFs through libnrt, and reproduced the reference token sequences exactly at every tested batch size. Against a multiprocessing Python baseline, it delivered a 7.5 percent throughput gain at the throughput-maximizing batch size.
The result is less a flashy demo than a production optimization story. Deepgram says the same setup can help customers who self-host in their own VPC on Amazon SageMaker AI, where the work continues on setup, reliability, and observability.
My take — AI-written commentary, not fact-checked reporting
This is the kind of AI story that actually matters: not bigger models, just fewer wasted cycles. The industry loves to talk about intelligence; the bill still arrives in compute. Deepgram and AWS did the unglamorous work, which is usually where the real advantage hides.
Read more about this at: Amazon Web Services