What happened
Artificial Analysis’s Speech to Text AI Model & Provider Leaderboard now includes a Diarization Index alongside its non-streaming word error rate, speed, and price comparisons. The index measures Word Diarization Error Rate, or WDER: the share of answer-key words not attributed to the person who said them. Lower is better. The leaderboard page labels the addition “New” and links to a methodology section describing the diarization evaluation.
The index is based on English conversations with between two and eight speakers. Artificial Analysis says it calculates the score as an equal mean across three speaker-count bands: two speakers, three to four speakers, and five to eight speakers. That design gives each band equal weight in the headline measure rather than letting the most common band dominate it. The leaderboard also presents results by dataset, including AA-Diarization and NOTSOFAR-1. For those dataset-level figures, it describes pooled WDER as wrong-speaker words plus missed words divided by scored reference words, without speaker-band weighting.
The displayed index contains 23 systems, with scores ranging from 3.3% to 20.1%. Nemotron 3.5 Transcribe is shown at 3.3%; Voxtral Small Transcribe 3 at 7.5%; Universal at 8.6%; and Soniox v5 Async at 20.1%. These are leaderboard results, not a general claim that one system will lead on every deployment’s recordings. Artificial Analysis also breaks out WDER by speaker count, giving buyers a view beyond the combined score when their application mostly handles pairs, small groups, or larger conversations.
Artificial Analysis places the diarization metric in a broader provider-comparison interface. Its non-streaming leaderboard also reports AA-WER, a word error rate index using AA-AgentTalk, VoxPopuli-Cleaned-AA, and Earnings22-Cleaned-AA, plus a speed factor and estimated price per 1,000 minutes of audio. New charts compare the Diarization Index with price and speed. The page explains that its speed factor is input-audio seconds transcribed per second of processing time, with values based on median benchmark trials from the last seven days using ten-minute audio. It describes price as an estimated cost normalized across provider billing models, including billed reasoning tokens where applicable.
The same page separately shows transcription accuracy for 47 of 83 models and API price and speed comparisons for 40 of 64 models, so coverage differs by measure. It notes that some models are chunked for the Earnings22 evaluation because of audio-length limits. The page links to a methodology explanation for the evaluation; the leaderboard itself does not state a release date or announce a new transcription model.
What the archive adds
New Runtime’s earlier coverage of Artificial Analysis’s Decision Index v1.0 described a benchmark focused on bounded model choices. The new diarization measure is a separate benchmark development, not a restatement: it applies evaluation to speaker attribution in speech transcription. An earlier New Runtime note argued that inspectable benchmark components make aggregate results more useful, while still leaving teams to judge whether the task mix fits their deployment.
Trend signal
This does not yet fit a pattern New Runtime tracks. The update does reinforce a general evaluation lesson: a single aggregate metric can obscure distinct failure modes. Here, word recognition and correct speaker assignment are presented separately, but one benchmark addition is not enough to establish a broader shift in agent or application architecture.
Why now
The leaderboard marks diarization results as new and adds comparisons against both price and speed. That makes this a timely update for teams choosing among speech APIs, particularly where identifying the speaker is part of the product rather than a later manual cleanup step.
Why it matters
Speaker attribution errors can change the meaning of a transcript. In a meeting summary, sales call record, interview archive, or support analytics pipeline, assigning a statement to the wrong person may be more damaging than a small number of misrecognized words. A separate WDER score makes that risk visible instead of treating all transcription errors as equivalent.
The comparison can help teams narrow a provider shortlist, but the ranking should be treated as a screening tool. The index covers English conversations and a defined range of speaker counts; local audio conditions, accents, overlap, domain vocabulary, and the consequences of attribution mistakes may differ. Teams should compare the leaderboard with a representative sample of their own recordings and inspect the actual output format and correction workflow before choosing a provider.
Limits and uncertainty
Artificial Analysis describes the index’s scoring and reports a leaderboard range, but benchmark results do not establish performance on every language, recording condition, or conversation type. The displayed index is limited to English conversations with two to eight speakers. A score aggregates across speaker-count bands and datasets, so it can hide cases where a provider performs unevenly. The page also presents different model counts for diarization, word error rate, price, and speed; these comparisons do not necessarily cover identical sets. Teams should validate results against their own labeled audio before treating rankings as expected production performance.
Reader takeaway
Artificial Analysis now makes speaker attribution errors visible as a separate speech-to-text comparison, rather than folding them into ordinary word accuracy. Use the Diarization Index to identify candidates, then test those candidates on representative conversations: the model that wins a leaderboard is not automatically the best choice for your speakers, audio, or workflow.