Lowest-Latency Inference APIs for Voice and Realtime Agents: A Time to First Token TTFT-First Benchmark
MarkTechPost Asif Razzaq
Voice latency isn’t just TTFT; the whole chain has to move fast. Some models look quick on paper and still feel slow when they finally speak.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Voice teams keep reaching for time to first token because it is easy to compare. And that’s the trap. For a chatbot, TTFT tells you a lot. For a voice agent, it only tells you when the engine starts turning over. A text-to-speech model still has to wait for a full clause before it can say anything, so the real user-facing metric is closer to time to first sentence, plus all the delay before that. LiveKit’s framing is blunt: TTFT is the right entry point and the wrong finish line.
The stack adds up quickly. LiveKit’s own overview puts a voice turn at roughly 100–200ms for speech-to-text, 300–500ms for the LLM with streaming, 100–200ms for text-to-speech, and 50–150ms for network over WebRTC. That lands the practical target around 700ms to 1.2s end to end. Kwindla Hultman Kramer, co-creator of Pipecat, has suggested aiming for 800ms median voice-to-voice latency, with 1,500ms as a looser proof-of-concept ceiling. Daily’s benchmark ties that to conversation itself: pauses beyond 800ms start to feel unnatural, and voice-to-voice under 1,500ms leaves roughly 700ms for the LLM in a transcription-to-LLM-to-voice setup.
On the model side, the numbers are all over the place, and the host matters as much as the weights. Artificial Analysis’ August 30, 2026 leaderboard shows Baseten’s gpt-oss-120b at 0.23s TTFT and DeepInfra’s Nemotron 3 Ultra at 0.28s, while Mercury 2 reaches 770 tokens per second but takes 3.07s to deliver its first chunk. Cerebras and Groq look much more voice-friendly because they combine strong TTFT with very high throughput. The same model can also behave differently depending on where it runs: GPT-5.6 Luna measured 0.59s on Amazon Bedrock and 0.74s on OpenAI’s own API.
LiveKit’s own numbers are even more pointed. It measured Gemma 4 31B at 192ms on its inference product, far ahead of Gemini 2.5 Flash at 911ms, GPT-5.5 at 966ms, GPT-4.1 at 1,006ms, and Gemma 4 31B through OpenRouter at 1,876ms. It says the trick is speculative decoding with SGLang and deliberately under-filled GPUs to keep queueing low. The tradeoff is cost, listed at $1.20 per 1M output tokens. Its reported time to first sentence for Gemma 4 31B on LiveKit was 354ms, versus 1,034ms for Gemini 2.5 Flash and 1,404ms for GPT-5.5.
The speech layers matter just as much. Deepgram Flux folds end-of-turn detection into recognition and says that can cut 200–600ms versus a separate VAD pipeline. ElevenLabs says Flash v2.5 is around 75ms for model inference, but also warns that network and buffering can add a lot more, with 500ms buffering common. Cartesia says Sonic-3.6 can do sub-90ms TTS. On the speech-to-speech side, Grok Voice Think Fast 2.0 High stands out at 0.70s TTFA, with 97% speech reasoning and 94.7% task success. The broader pattern is clear: the fastest voice agents are built by shaving every stage, not by worshipping a single sexy latency number.
My take — AI-written commentary, not fact-checked reporting
The industry still loves TTFT because it’s a tidy headline number, which is adorable in the way spreadsheets are adorable. For voice, the winning move is boring: shorten the sentence path, trim queueing, and stop pretending a fast first token means a fast conversation. Open models keep looking better where real control matters, and the hosted stacks keep proving that routing and buffering can ruin the party all by themselves.
Read more about this at: MarkTechPost