**Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**

NVIDIA has put out an open-weight diarization model that tries to do the annoying part of conversation AI: figure out who spoke when, even when people overlap. Nemotron 3 Diarization is a 100M-parameter model, supports up to eight speakers, and ranked first on VoiceArena’s Diarization-Bench with a 14.72% diarization error rate. That matters because transcripts without speaker labels are only half useful. You can read the words. You still don’t know who said them.

The model is built for both recorded audio and streaming use, which is where diarization gets messy fast. In a live conversation, the system only sees a chunk of audio at a time, so it has to keep speaker labels stable across pauses, interruptions, and long gaps. NVIDIA says Nemotron 3 uses arrival-order labeling: the first new voice becomes speaker 1, the next new voice becomes speaker 2, and so on. That avoids having to re-solve speaker identity every chunk. The design also handles overlapping speech, so two speakers can be active at once.

Under the hood, the model takes 16 kHz mono audio, turns it into Mel-spectrogram features, and feeds them through a 31-layer Transformer encoder. The output is a [T, 8] tensor of speaker-activity probabilities, one channel for each possible speaker. During streaming inference, it uses an Arrival-Order Speaker Cache and a FIFO queue to keep context. NVIDIA says the model can run at different latency points, with recommended input-buffer settings of 30.4, 1.04, 0.64, and 0.32 seconds. Shorter buffers respond faster; longer ones usually improve accuracy.

The benchmark story is the bigger one. On VoiceArena’s initial Diarization-Bench, Nemotron 3 ranked first among 12 systems and 17 configurations across 139 English-language conversations totaling about 22 hours. It posted 14.72% DER, ahead of the next-ranked system at 19.3%. NVIDIA also says the model was evaluated across 901 recordings covering multilingual telephone speech, meetings, near-field and far-field microphones, multi-microphone capture, and other difficult audio. In those tests, it cut DER versus NVIDIA’s earlier four-speaker Streaming Sortformer baseline at every shared latency point.

There is still a catch, because there always is. The company says performance can slide on very long recordings or on noisy, reverberant, far-field, or domain-shifted audio. And diarization is only one half of the transcription stack; ASR still has to supply the words. But for anyone building speaker-attributed transcripts, call analytics, or voice-agent memory, this is the useful kind of progress: less demo magic, more channel bookkeeping.