Stream low-latency speech from AI agents using Kokoro TTS on Amazon SageMaker AI
Amazon Web Services Joseph Cottingham
AWS says it found a way to start voice playback before an AI finishes its full answer. The trick is streaming sentences into Kokoro TTS on SageMaker instead of waiting for the whole reply.
Based on reporting by Amazon Web Services, Joseph Cottingham — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is pitching a fix for one of the most annoying delays in voice agents: the awkward silence between an AI starting to think and a user actually hearing anything. The company says a typical one-sentence answer can generate in about 600 milliseconds, while a longer response may take five seconds before text-to-speech even starts. That gap is the problem. The solution is to feed each sentence to Kokoro-82M on Amazon SageMaker AI as soon as it appears, so synthesis begins on sentence one rather than after the full response is done.
Kokoro-82M is a small model by modern standards, with 82 million parameters. AWS runs it in a custom container on an ml.g5.xlarge GPU instance, behind a SageMaker AI real-time endpoint. The container exposes two routes: a standard POST route and a bidirectional WebSocket route. The point of the setup is simple enough: keep latency down, keep inference cost down, and avoid paying for a bigger machine than the workload needs.
The post spends a lot of time on transport details, because that’s where these systems usually fall apart. SageMaker’s bidirectional streaming sends client messages as binary WebSocket frames, so a Starlette handler that only reads text will break even though the endpoint looks healthy. The connection also has to stay open across multiple messages; closing after each sentence would throw away most of the latency win. AWS’s incremental sentence chunker then turns either a full string or a live token stream into complete sentences, sending each one as soon as it’s ready.
That does not mean speech synthesis is happening in parallel. It’s still a sequential loop: receive a message, synthesize it, send the audio, mark completion, then wait for the next message. The gain comes from starting sooner, not from overlapping every step. AWS says it tested this with a live endpoint and got audio back for each sentence in a three-sentence input, with matching chunk sizes and durations for each piece.
The other half of the problem is getting audio back to the browser. If a team already uses Server-Sent Events, AWS says base64-encoding audio inside SSE is the practical route, even though it adds about 33% overhead and needs buffering because audio frames and SSE chunks don’t line up neatly. If the voice path is being built from scratch, the cleaner option is to keep raw binary audio and JSON control frames on the same bidirectional session, which avoids the extra encoding step. In the end, the post is less about one model than about the plumbing that keeps a voice agent from sounding slow and clumsy.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of boring: real engineering, not another demo that claps when the slide deck says “low latency.” The industry keeps pretending the model is the product, when half the user experience lives in framing, buffering, and not closing the socket like an amateur. Voice agents don’t need more hype. They need fewer five-second pauses.
Read more about this at: Amazon Web Services