The initial index covers DeepSeek V4.1 Flash, Kimi K3, and MiniMax M3 through their first-party APIs and selected third-party providers.
Each model-provider comparison uses 165 tasks drawn from finance, coding, and US legal research: 66 Finance Agent v2 tasks, 33 Vibe Code Bench tasks, and 66 Legal Research tasks. Vals records accuracy, cost per successful task, response timing, generation speed, and tool-call failures. It says it will rerun the index every two weeks with a fresh first-party API baseline.
The published results show endpoint-level cost and speed variation. Kimi K3 cost $12.65 per task through Baseten, versus $5.43 to $5.97 through Together AI, Fireworks, and Moonshot. MiniMax M3 cost $1.01 through Together AI and $1.07 through Fireworks, compared with $2.18 through MiniMax. For DeepSeek V4.1 Flash, overall accuracy differences were within reported error bars.
The source does not establish a universal provider ranking or results for models and workloads outside this set. Its measurements depend on the selected tasks, settings, provider availability, and the time of each run. Source: https://www.vals.ai/benchmarks/inference_provider