Announcing the fastest inference for realtime voice AI agents

Building a voice agent has always meant stitching together a transcription API from one vendor, a text-to-speech service from another, and hoping the latency between them doesn't wreck the conversation. Together AI is betting that developers are tired of that assembly work, and it just shipped a set of audio tools meant to collapse the whole pipeline into one cloud.

The headline piece is streaming Whisper, which Together claims completes transcripts up to 35% faster than competing services by running over WebSocket connections instead of constantly reopening them, paired with voice activity detection tuned to catch the exact moment someone stops talking. That detail matters more than it sounds. Cut a user off too early and the agent looks rude; wait too long and the silence reads as broken. Together says it optimized for time-to-complete-transcript rather than just the flashier time-to-first-token metric, which is really an admission that a lot of the industry has been optimizing for the wrong number.

On the output side, Together is now the first cloud offering serverless open-source text-to-speech, with two models aimed at opposite ends of the spectrum. Orpheus targets natural, expressive speech at 187ms time-to-first-byte, positioned as good enough for customer-facing brands that care how they sound. Kokoro strips things down to 97ms TTFB for high-volume deployments where speed beats polish, roughly twice as fast as rival offerings according to Together's own numbers. Neither requires spinning up dedicated GPU instances, which has been the annoying default for anyone wanting TTS without committing to constant infrastructure spend.

For batch work, Together added Voxtral Mini from Mistral AI, a transcription model tuned for European languages and messy audio, plus automatic speaker diarization that labels who said what in a recording. Together frames these as the accuracy tier for situations where a wrong word carries real consequences: medical notes, legal depositions, financial calls. It's the same logic Whisper users hit constantly — cheap and fast is fine until the stakes go up, then you pay for precision.

The pitch underneath all of this is architectural, not just feature-based. Together is running these models on the same GPU clusters as its LLM inference, avoiding cross-provider network hops, and multiplexing multiple live conversations through single WebSocket connections so a contact center running hundreds of simultaneous calls doesn't need thousands of open sockets. Whether that scales as gracefully in production as it does in Together's own benchmarks is the part outside vendors will actually test.