---
type: "post"
id: "introducing-frontierharness-eval-v1-1-fd98d9"
slug: "introducing-frontierharness-eval-v1-1-fd98d9"
title: "Introducing FrontierHarness Eval v1.1"
description: "Runta and Baseten’s FrontierHarness Eval v1.1 compares 22 agent-harness configurations across five models and 62 terminal tasks."
retrieval_nugget: "Runta and Baseten’s FrontierHarness Eval v1.1 compares 22 agent-harness configurations across five models and 62 terminal tasks. Its results suggest that harness cost rankings can change sharply with the model, while repeated agent steps—not simply prompt size—track cost more closely. Teams should evaluate the model-harness pair on their own work."
published_at: "2026-11-19"
updated_at: "2026-11-19"
record_date: "2026-11-19"
date_kind: "published_at"
topics: ["codex"]
entities: ["Runta","Baseten","FrontierHarness Eval","Harbor","Codex CLI","GLM-5.3","Kimi-K3","OpenCode"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: null
prior_refs: []
source_url: "https://runta.com/blog/introducing-frontierharness-eval-v1-1"
source_urls: ["https://runta.com/blog/introducing-frontierharness-eval-v1-1"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:introducing-frontierharness-eval-v1-1-fd98d9"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/introducing-frontierharness-eval-v1-1-fd98d9/","markdown":"https://newruntime.com/posts/introducing-frontierharness-eval-v1-1-fd98d9.md","json":"https://newruntime.com/posts/introducing-frontierharness-eval-v1-1-fd98d9.json"}
markdown_url: "https://newruntime.com/posts/introducing-frontierharness-eval-v1-1-fd98d9.md"
---

# Introducing FrontierHarness Eval v1.1

## Retrieval answer

Runta and Baseten’s FrontierHarness Eval v1.1 compares 22 agent-harness configurations across five models and 62 terminal tasks. Its results suggest that harness cost rankings can change sharply with the model, while repeated agent steps—not simply prompt size—track cost more closely. Teams should evaluate the model-harness pair on their own work.

## Summary
Runta and Baseten’s FrontierHarness Eval v1.1 compares 22 agent-harness configurations across five models and 62 terminal tasks. Its results suggest that harness cost rankings can change sharply with the model, while repeated agent steps—not simply prompt size—track cost more closely. Teams should evaluate the model-harness pair on their own work.

## What happened
Runta’s November 17, 2026 announcement describes v1.1 of FrontierHarness Eval, a benchmark built with Baseten. The earlier v1.0 leaderboard tested nine harnesses on one model; v1.1 expands the evaluation to 22 configurations, five models, 62 tasks and four attempts per task. Runta reports 27,280 agent runs and 83 billion tokens. The 22 configurations count multiple profiles of DeepSeek Harness and two OpenCode versions separately. All five models were served by Baseten: GLM-5.3, GLM Flash, DeepSeek V4.1 Flash 1017, Muse and Kimi-K3.

The tasks were terminal-based, with time limits ranging from 15 minutes to eight hours. Runta used Harbor to evaluate each harness with its default settings, holding each harness version and settings constant across models and using maximum reasoning effort. Every trial received a fresh Runta runtime that was deleted afterward, preventing access to earlier trial files. The authors calculate average cost per task from priced trials and use cost per pass—average task cost divided by pass rate—to select a “best-value” harness. Their selection rule allows a harness that costs up to 30% more per pass to qualify if it passes at least five percentage points more often.

The best-value pairing varied by model: Codex CLI with GLM-5.3, Pi with GLM Flash, ZCode with DeepSeek, Oh My Pi with Muse, and Alma with Kimi-K3. Across all models, OpenCode v2 had the lowest average cost per task at $0.561, while ZCode recorded the highest pass rate, 61.9%. Codex CLI ranked first for cost with GLM-5.3 but sixteenth with Kimi-K3. Runta says the median harness moved 11 of 22 cost positions between models. With a model held fixed, the most expensive harness cost 1.8 to 5.0 times as much per task as the cheapest. The benchmark’s resampling analysis found Codex CLI remained best value with GLM-5.3 in 99.6% of 4,000 resamples; the other model-specific winners were less certain.

Runta also examined factors associated with cost. Across harnesses, steps per pass correlated with cost per pass (+0.51, p=0.015); prompt size, tool count and cache-hit rate did not show a comparable relationship. Each step resends context, and trials used an average of 2.0–4.8 million input tokens per harness. Of the list-price cost for scored runs, 64.5% went to trials that failed or timed out. The five best-value harnesses shared observed design traits, including concentrating most tool calls in one shell tool, but Runta presents these as observations rather than tested causes.

## What the archive adds
New Runtime’s earlier coverage on August 26, 2026, [“Agent work is moving into harnesses, factories, and accepted artifacts”](https://newruntime.com/posts/agent-work-is-moving-into-harnesses-factories-and-accepted-artifacts/), described a broader shift toward harnesses as an important layer of agent infrastructure. This is a new benchmark result on that known theme: it quantifies how harness choice and model choice interact on cost and pass rates.

## Trend signal
This supports [“Harness architecture outlives model choice”](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/), with a useful qualification. Runta finds that the harness-model pairing changes both economics and observed task performance, so a model switch may require retesting the surrounding harness. The benchmark does not establish that harnesses are interchangeable or that one configuration wins across providers and tasks.

## Why it matters
Teams often compare models using a fixed agent setup, or compare harnesses on a single preferred model. Runta’s results show why either shortcut can mislead: Codex CLI moved from first to sixteenth in cost rank across two models, and harness choice changed cost by several multiples when the model stayed fixed. A lower-cost configuration is not automatically a better choice either; pass rates and uncertainty matter, and Runta found no consistent relationship in which higher cost bought higher pass rates.

The cost distribution also makes failed work part of the operating model, not an edge case. If almost two-thirds of scored-run cost goes to failures and timeouts, teams need to track cost per accepted outcome alongside token prices and successful-run averages. The study further suggests that repeated steps deserve scrutiny: reducing prompt size alone may not reduce total spend when each iteration resends a large context.

## Limits and uncertainty
The benchmark uses open models served by one provider. Runta says closed models may produce different results, and Baseten’s prices and caching affect the cost rankings. The task set is 62 terminal tasks, so the results do not establish how harnesses compare on other work, longer-running agents or production workloads. Several model-specific best-value results are close: the authors report confidence intervals of about plus or minus ten percentage points for pass rates, making small differences uncertain. The observed traits of the leading harnesses were not tested as causal interventions. Runta also says a stop-time check might reduce wasted cost, but did not test that proposal.

## Skill to build now
When you change a model or harness, rerun a small, representative evaluation rather than carrying forward the old ranking. Use tasks drawn from your own workload, repeat them enough to see whether close pass-rate differences hold up, and record cost per task, pass rate and cost per pass separately. Include failed and timed-out runs in the cost total. For agent developers, inspect step counts and tool-call traces before focusing only on the initial prompt’s token count.

## Reader takeaway
Treat the model and harness as a pair to evaluate, not independent choices. Runta’s benchmark shows that the cheapest or best-value harness can change with the model, and repeated steps can matter more to cost than prompt size. Validate the pairing on your own tasks, and account for the cost of runs that do not pass.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Codex](https://newruntime.com/topics/codex.md): Explore the codex topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
