A team from Hume published a study on 21 August with an uncomfortable premise: public voice AI benchmarks increasingly suggest models perform at human level, while those scores do not always reflect how the models behave in the real world. Because public benchmarks are open and widely used, models can become optimized for the tests themselves, so a score can improve because the model learned benchmark-specific patterns rather than because it got better at the task.
The response is structural rather than rhetorical. The authors introduced held-out sets in Real World VoiceEQ, the Open-ASR Leaderboard and the Far-field ASR Leaderboard, so a public score can be compared against data the model could not have trained on. They also argue traditional benchmarks overlook conditions that make voice systems reliable, natural and contextually appropriate in practice, which means the gap is not only contamination but coverage.
Moving integrity into the measurement design is the same correction as double-blind evals moving benchmark integrity into the run, and it is why evals move from leaderboards to operating diagnostics and why a minimum integrity stack matters before a ranking is quoted.
The analysis comes from a vendor that competes in voice AI and publishes the leaderboards in question, and held-out performance is reported rather than independently audited. The condition to watch is whether held-out and public scores are published side by side for every entrant, because a private set only removes contamination if nobody gets a second attempt at it.