How Together AI built the world’s fastest speech-to-text stack

Together AI · 29 May 2026, 00:00

Together AI optimized its speech-to-text serving stack to transcribe 20 hours of audio in under 10 seconds using NVIDIA's Parakeet-TDT 0.6B v3 model. Key improvements included multi-profile TensorRT compilation for variable audio lengths, moving decoder branch logic onto GPU with conditional CUDA graphs for 2-3x faster decoding, eliminating redundant CPU copies through shared memory and Unix domain sockets, using epoll for streaming connection handling, and calling gc.freeze() to prevent garbage collection pauses. The changes address the full data path from preprocessing through GPU inference, reducing latency and tail latency spikes in production voice agent systems.

Why it matters

Together AI built the fastest speech-to-text stack on Artificial Analysis by treating ASR as a full-path systems problem, not just a GPU inference problem.