NVIDIA Releases Nemotron 3 Diarization: A 100M-Parameter Open-Weight Model That Tracks 8 Speakers in Real Time

NVIDIA has put out Nemotron 3 Diarization on Hugging Face, and the pitch is refreshingly specific: this model answers the “who said what, when” problem that speech recognition leaves behind. It is a 100M-parameter open-weight system, and the license allows commercial use. That matters because diarization is one of those unglamorous steps that decides whether a transcript is actually useful or just a wall of words with no owner.

The model tracks up to 8 speakers, including overlap, which is a meaningful jump from NVIDIA’s earlier 4-speaker Streaming Sortformer checkpoint. It uses a single checkpoint for both offline recordings and streaming. Under the hood, it takes 16 kHz single-channel audio, turns it into Mel-spectrogram features with a 10 ms step, stacks them by 8, and feeds the result through a 31-layer Transformer encoder with rotary positional embeddings. A Conv1D layer then lifts the predictions back to 10 ms resolution.

The output is a [T, 8] set of activity probabilities, so multiple voices can be active in the same frame. NVIDIA also keeps the speaker labels stable across chunks by ordering voices by arrival time and using an Arrival-Order Speaker Cache plus a FIFO queue for recent context. The labels stay anonymous; anything that connects them to real people has to happen downstream.

NVIDIA gives four latency settings, from a 30.4-second offline-style buffer down to 0.32 seconds at the low end. On DIHARD III, DER ranges from 12.73% at the offline-style setting to 13.55% at ultra-low latency, and the reported throughput reaches 15,113× RTFx in the batched offline-style test on an RTX PRO 5000 with torch.compile(). The company is careful to say those are batched numbers, not end-user latency.

Benchmarking is where the model starts sounding less like a lab demo and more like a serious release. On Voice Arena’s initial Diarization-Bench, it ranked first among 12 systems and 17 configurations, scoring 14.72% DER against 19.3% for the next-best system. NVIDIA also says that, compared with its 4-speaker baseline at 1.04 seconds of latency, Nemotron 3 improved across all 8 evaluation conditions, with an average relative reduction of 41.0%.

There are limits, and NVIDIA says them plainly. More than 8 speakers can lead to missed or misassigned speech, and heavy noise, reverberation, or far-field audio can push error rates up. For a model meant for meeting tools, call analytics, podcasts, and voice-agent memory, that honesty is probably the most useful feature after the speaker tracking itself.