---
type: "post"
id: "vals-ai-launched-the-inference-provider-index-comparing-60fbd1"
slug: "vals-ai-launched-the-inference-provider-index-comparing-60fbd1"
title: "Vals AI launched the Inference Provider Index comparing open-weight model endpoints on agentic tasks."
description: "Vals AI has published the Inference Provider Index, a comparison of endpoints serving the same open-weight models on multi-step agent tasks."
retrieval_nugget: "Vals AI has published the Inference Provider Index, a comparison of endpoints serving the same open-weight models on multi-step agent tasks."
published_at: "2026-10-24"
updated_at: "2026-10-24"
record_date: "2026-10-24"
date_kind: "published_at"
topics: ["agent-economics","model-routing"]
entities: ["Vals AI","Inference Provider Index","DeepSeek V4.1 Flash","Kimi K3","MiniMax M3","Together AI","Fireworks","Baseten"]
related_patterns: ["agent-economics-moves-to-completed-work","harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "Vals AI has launched a separate index comparing inference providers on agentic tasks, finding up to 2.3× cost differences for the same model and that first-party APIs were neither cheapest nor fastest."
prior_refs: ["memory:8e300951-d712-8cd6-a209-8b4bcba4f2e6","memory:8e27a4a1-85d0-8284-a85b-06f3f8600b47"]
source_url: "https://www.vals.ai/benchmarks/inference_provider"
source_urls: ["https://www.vals.ai/benchmarks/inference_provider"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:vals-ai-launched-the-inference-provider-index-comparing-60fbd1"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/vals-ai-launched-the-inference-provider-index-comparing-60fbd1/","markdown":"https://newruntime.com/posts/vals-ai-launched-the-inference-provider-index-comparing-60fbd1.md","json":"https://newruntime.com/posts/vals-ai-launched-the-inference-provider-index-comparing-60fbd1.json"}
---

# Vals AI launched the Inference Provider Index comparing open-weight model endpoints on agentic tasks.

## Retrieval answer

Vals AI has published the Inference Provider Index, a comparison of endpoints serving the same open-weight models on multi-step agent tasks.

Vals AI has published the Inference Provider Index, a comparison of endpoints serving the same open-weight models on multi-step agent tasks. The initial index covers DeepSeek V4.1 Flash, Kimi K3, and MiniMax M3 through their first-party APIs and selected third-party providers.

Each model-provider comparison uses 165 tasks drawn from finance, coding, and US legal research: 66 Finance Agent v2 tasks, 33 Vibe Code Bench tasks, and 66 Legal Research tasks. Vals records accuracy, cost per successful task, response timing, generation speed, and tool-call failures. It says it will rerun the index every two weeks with a fresh first-party API baseline.

The published results show endpoint-level cost and speed variation. Kimi K3 cost $12.65 per task through Baseten, versus $5.43 to $5.97 through Together AI, Fireworks, and Moonshot. MiniMax M3 cost $1.01 through Together AI and $1.07 through Fireworks, compared with $2.18 through MiniMax. For DeepSeek V4.1 Flash, overall accuracy differences were within reported error bars.

The source does not establish a universal provider ranking or results for models and workloads outside this set. Its measurements depend on the selected tasks, settings, provider availability, and the time of each run. Source: https://www.vals.ai/benchmarks/inference_provider

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agent Economics](https://newruntime.com/topics/agent-economics.md): Explore the agent-economics topic hub.
- topic hub: [Model Routing](https://newruntime.com/topics/model-routing.md): Explore the model-routing topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)