First implication: Teams evaluating routers should ask whether evidence captures task success, total spend, and session-level effects together, instead of relying on a single-call benchmark or provider-specific result.
First watch condition: Watch for Vals to publish results from the shared-harness comparison, including task domains, router configurations, quality measures, and total costs.
Thesis
Model routers should be judged by verified work across long-running agent sessions, not only by their accuracy on isolated prompts.
Observed facts
Vals reports that five different models occupy the top positions across its eight benchmarks, and that no model ranks in the top three on all of them.
1It calculates that an oracle router picking the best model for every task in hindsight would beat the best single model on the Vals Index by 14 points at about 40% lower cost.
1
Mechanism
A routing choice changes more than the price of one request. In a multi-step agent workflow, a weaker answer can trigger recovery work or alter later choices; switching models can also sacrifice prompt-cache savings.
Why now
Vals says ten developers released 16 language models tracked by its index in the past month alone, increasing the selection burden for teams.
Implications
Teams evaluating routers should ask whether evidence captures task success, total spend, and session-level effects together, instead of relying on a single-call benchmark or provider-specific result.
Evidence boundary
The article describes Vals’ argument and planned comparison, alongside its own index and oracle calculation. It does not provide results from the proposed independent router comparison, and the oracle assumes hindsight-perfect choices that cannot be reproduced operationally.
Watch conditions
Watch for Vals to publish results from the shared-harness comparison, including task domains, router configurations, quality measures, and total costs.