PolyAI Releases Dialog-RSN-1: An Audio-Native Dialog Model That Fuses Turn-Taking, Speech Recognition, Function Calling, And Response

Most voice AI still works like a game of telephone: audio gets converted to text, the text goes to an LLM, and everything about how something was said—hesitation, tone, that half-second pause before someone changes their mind—gets thrown away before the model ever sees it. PolyAI's new Dialog-RSN-1 skips that step. It listens to the actual audio, decides whether the caller has finished talking, generates a response or triggers a function call, and only afterward runs transcription in parallel with speech generation. Transcription becomes a byproduct rather than the bottleneck.

The turn-taking trick is clever. The model's very first output token is one of three states: EMPTY, ONGOING, or COMPLETE. Rather than running constantly and burning GPU time on every call like a full-duplex streaming model, Dialog-RSN-1 sits idle until a lightweight voice-activity detector and a couple of timers flag that something worth checking just happened. Cheap signals decide when to ask; the model, with full context, decides what to do about it. That's a meaningfully different architecture than the always-on speech-to-speech systems from Google and OpenAI, which pin a GPU for an entire call and bake the output voice into the model itself, limiting how much control you get over pronunciation.

PolyAI kept text-to-speech separate on purpose, which sounds like a small detail but isn't. It means enterprises can still control exactly how the agent sounds while getting the benefits of audio-native understanding on the input side. The company post-trained open-weight multimodal models—testing Gemma, GPT-OSS, Qwen, and Mistral—with supervised and reinforcement finetuning on its own call data, landing on models in the 8B-dense-to-30B-sparse range to hit sub-300ms response times on A100s. The engineering underneath includes prefilling attention caches while the caller is still speaking, an append-only prompt format to avoid cache invalidation, and a speculative drafter that accepts an average of 3.9 tokens at a time.

The numbers PolyAI is citing are specific enough to take seriously: an 11% relative bump in containment for a restaurant chain, a 37% latency drop for an insurer, and internal benchmarking (on a self-built eval called Dialog-Eval, which PolyAI says it'll open-source) showing Dialog-RSN-1 outscoring cascaded pipelines that top out around 77 on an audio-awareness score. Transcription quality also improved as a side effect—gpt-4o-transcribe's word error rate dropped from 7.8% to 6.9% with added context, and PolyAI says its own model does better still.

None of this is available as open weights or a public API, though. It's English-only for now, and it ships exclusively through PolyAI's platform to existing enterprise customers, with early access available to new ones. This is squarely built for large call-center operations in insurance, healthcare, hotels, and similar high-volume industries—not for a developer who wants to bolt voice AI onto a side project. PolyAI closed an $86 million Series D in December, and it says it already has more than 100 enterprise customers running 2,000-plus live deployments, so the audience for this was never really in question.