How Together AI built the world’s fastest speech-to-text stack

Together AI · 1 month ago

Together AI optimized its speech-to-text serving stack to transcribe 20 hours of audio in under 10 seconds using NVIDIA's Parakeet-TDT 0.6B v3 model. Key improvements included multi-profile TensorRT compilation for variable audio lengths, moving decoder branch logic onto GPU with conditional CUDA graphs for 2-3x faster decoding, eliminating redundant CPU copies through shared memory and Unix domain sockets, using epoll for streaming connection handling, and calling gc.freeze() to prevent garbage collection pauses. The changes address the full data path from preprocessing through GPU inference, reducing latency and tail latency spikes in production voice agent systems.