How Together AI built the world’s fastest speech-to-text stack
Together AI
Together AI rebuilt its speech-to-text pipeline top to bottom and now claims the fastest transcription stack around. They squeezed the gains not from the AI model itself but from fixing the boring plumbing everyone ignores.
Based on reporting by Together AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Speech-to-text sounds like a smaller cousin of the LLM problem, but Together AI's engineering writeup makes a decent case that it's actually a harder systems puzzle. Text prompts are compact and arrive nearly ready for a GPU. Audio doesn't work that way — the same content as audio can be a thousand times bigger, and before a single byte reaches the model it has to be decoded, resampled, denoised, segmented, and turned into features. Because ASR models like NVIDIA's Parakeet-TDT 0.6B v3 and OpenAI's Whisper Large v3 are comparatively tiny, in the hundreds of millions to low billions of parameters, all that surrounding data-handling work matters far more than it would for a giant LLM.
Together's fix started with the encoder, which holds about 95% of Parakeet's weights. Instead of one generic execution plan, they compiled multiple TensorRT profiles tuned to the actual range of audio lengths they see in production, from 200-millisecond streaming chunks up to 30-second clips. That avoided forcing short snippets through a plan built for long audio, which the team says made small inputs several times slower than they needed to be.
The decoder turned out to be the sneakier bottleneck. Each loop iteration only did microseconds of GPU work, but a simple if-token-is-blank check forced a round trip to the CPU every single time, which blocked the whole thing from running as one efficient CUDA graph. Swapping that check for a conditional CUDA graph node let the branch resolve entirely on the GPU, cutting decoder time by two to three times. Similar unglamorous fixes showed up elsewhere: replacing chatty microservice hops and ZeroMQ with raw Unix domain sockets and shared memory to kill redundant data copies, and switching streaming connections from one-thread-per-socket to a single thread parked on epoll so hundreds of simultaneous streams don't trigger a scheduler stampede.
The strangest bug lived in Python itself. Latency was fine at the median but p95 would randomly spike by roughly 200 milliseconds, and it turned out the culprit was garbage collection walking through a huge pool of buffers and lookup tables the team had preallocated at startup. One line, gc.freeze(), told Python to stop scanning that long-lived state, and the spikes vanished.
Put together, the changes let Parakeet-TDT 0.6B v3 transcribe about 20 hours of audio, roughly the length of the entire Harry Potter film franchise, in under 10 seconds. Together frames this as evidence that voice AI latency is won or lost across the whole pipeline — GPU kernels, CPU preprocessing, transport, and runtime internals — not just inside the model.
My take — AI-written commentary, not fact-checked reporting
This is the kind of unglamorous engineering that never trends but actually moves products, and it's a good reminder that model benchmarks mean little if your socket handling and garbage collector are quietly adding 200ms of jitter. Voice AI's real bottleneck was never going to be parameter count, it was always going to be plumbing, and Together just proved it with receipts.
Read more about this at: Together AI