NVIDIA Releases NemotronLabs VoiceChat 11B: An Open Full-Duplex Speech-to-Speech Model with ~450 ms Turn-Taking and Live Tool Calling
MarkTechPost Asif Razzaq ● Covered by 2 sources
NVIDIA released an open 11B voice model that can listen and talk at the same time. It also does live tool calls without going silent, but NVIDIA still says it’s research-only.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
NVIDIA has put out NemotronLabs VoiceChat 11B, an open 11B speech-to-speech model built for real-time, full-duplex conversation. The pitch is simple: one model handles the whole exchange, instead of stitching together speech recognition, a language model, and text-to-speech. That matters because every handoff adds delay, and voice systems get awkward fast when they can’t keep up with a human interrupting them.
On Full-Duplex-Bench 1.0, NVIDIA says the model reached 448 ms for smooth turn-taking, with a take-over rate of 1.00 at 480 ms for user interruptions. In plain terms, it can keep listening while it speaks, then yield when someone jumps in mid-turn. That is the kind of behavior voice agents need if they’re supposed to sound less like phone menus with opinions.
The bigger twist is tool calling. NVIDIA says this is the first open full-duplex model that can call tools while the conversation keeps flowing, using a separate output channel for <TOOLCALL> scripts and operator-defined on-hold lines while an API runs. The model outputs agent audio, agent text, and a running user transcription, so the conversation is supposed to stay alive even when the system is waiting on something else.
But the release notes keep the brakes on. The weights and container are public, the license is permissive, and NVIDIA says the model is for research purposes only. The repo also documents some ugly failure modes: a two-minute audio context ceiling, gibberish after several turns, runaway self-talk after a turn ends, and missing words in transcription. So yes, it’s deployable for pilots. No, it does not read like something anyone should quietly slip into production.
The hardware bar is also blunt: one GPU with at least 80 GB of VRAM, such as an A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. There’s no hosted API, and no inference provider is serving it yet, which means teams without GPU access can’t even kick the tires. NVIDIA positions it for contact centers, in-cabin assistants, retail ordering, telecom IVR, games, and accessibility work, all of which are exactly the places where low latency and barge-in behavior matter most.
My take — AI-written commentary, not fact-checked reporting
Open voice models are getting better at the hard part now: not the demo, the timing. But NVIDIA still did the honest thing here by saying “research only” instead of pretending a flashy benchmark makes production safe. That restraint is rare enough to deserve a nod; the rest of the industry still loves shipping a lab result in a blazer.
Read more about this at: MarkTechPost
