Baseten says NVIDIA Nemotron 3 Diarization is a roughly 100M-parameter transformer model that converts 16 kHz audio into 10 ms speaker-activity frames for as many as eight speakers. It also says one checkpoint can operate across four algorithmic latency profiles from 0.32 to 30.4 seconds. The supplied release reports a 9.8% diarization error rate on AISHELL-4 at its “low” profile, compared with 27.2% for Streaming Sortformer v2.1 in that test.
The mechanism is a streaming architecture that maintains an arrival-order speaker cache and a FIFO of recent frames, avoiding embedding-based clustering for an unbounded conversation. That gives the application an ongoing per-speaker activity signal; Baseten says it can feed VAD, speaker-specific end-of-turn detection, interruption detection, overlap handling, and a paired Parakeet V2 transcription flow.
Why now is that voice-agent systems increasingly need to decide when a user has finished, whether another person is speaking, and whether an interruption should change the agent’s behavior. A diarization component that emits labels 300 ms to 1 s behind live audio could make those decisions part of the runtime path instead of a later transcript-cleanup task.
The immediate implication is that voice-stack builders can evaluate a simpler integrated path for speaker-aware endpointing and attributed transcription, with a latency setting selected per request. A second implication is that throughput economics may improve if diarization and ASR share timing and activity features rather than requiring separately aligned pipelines.
Evidence boundary: the latency, quality, throughput, and cost framing come from Baseten’s own product release and benchmarks, not an independent evaluation. The source reports more than 500 concurrent hour-long diarization streams per RTX PRO 6000 in its optimized setup, but this does not establish equivalent capacity or accuracy for every acoustic environment, language, model host, or application design.
Watch whether independent deployments reproduce low-latency diarization quality on noisy, overlapping, multi-party calls, and whether the per-speaker activity signal measurably improves voice-agent endpointing and interruption behavior. Also watch whether the claimed integration advantage persists when teams combine the model with ASR systems other than the referenced Parakeet V2 pipeline.