What happened
The Hugging Face Space presents itself as a way to answer which coding-agent harness to run with an open model. Its current version is v0.1.1. The leaderboard covers nine harnesses: Claude Code 2.1.300, Codex 0.162.1, Pi 1.2.0, OpenCode 1.19.0, Hermes Agent v0.22.1, mini-swe-agent 2.4.6, Kimi Code 2.2.0, Qwen Code 0.25.2, and OpenClaw 2026.10.4. Each harness is run on the same models, provider, and reasoning effort, with three attempts per task, model, and effort.
The evaluated models are Flint, DeepSeek-V4.1-Flash, and GLM-5.3-Flash. The benchmark combines Terminal-Bench 2.1, scoring 87 of 89 tasks, and DeepSWE v1.1, scoring 111 of 113. Runs take place on Hugging Face: inference providers serve models where available, while Flint uses a dedicated inference endpoint; trials run in individual HF Sandboxes, with HF Jobs and Harbor coordinating them. Results and trajectories are published in a public bucket.
The board reports mini-swe-agent as the top overall harness at 51.4% macro average, with a 95% confidence interval of 48.5%–54.2%. Kimi Code, Hermes Agent, Pi, and Claude Code are statistically tied with that leader under the board’s paired-bootstrap test. Pi is listed at $0.094 per trial and $0.29 per solved trial; mini-swe-agent is $0.17 per trial and $0.53 per solve. The lowest macro score is OpenClaw’s 27.3%. The board says costs are calculated from reported tokens and provider list prices, except dedicated endpoints, whose token price is estimated from a load test.
The leaderboard also compares harness effects with run-to-run variation. Repeating the same harness flips the outcome on 23.4% of tasks; swapping harnesses flips 29.1%. The board interprets the 5.7-point difference as evidence that harness choice adds variation beyond rerun noise. Infrastructure failures before an agent can work are retried up to twice and excluded; agent errors and timeouts count as failures. Cheating, including looking up solutions or calling another model, counts as failure, though some trajectory checks are incomplete. DeepSWE runs offline since October 23, 2026, but earlier trials that remain in the results mix online and offline runs.
What the archive adds
New Runtime’s earlier coverage argued that harness choice is a model- and task-dependent control surface, not a fixed wrapper, in its HarnessTax analysis (October 22, 2026). A later benchmark note (November 6, 2026) found that asking Claude to think harder beat several harness setups. This leaderboard is a new measurement on open models, extending a known story with a controlled comparison and public trajectories.
Trend signal
This supports the view that harness architecture can be a durable control surface independent of model choice. The board’s same-model comparisons show meaningful differences among harnesses, although its top five are statistically tied and run-to-run noise remains substantial. See Harness architecture outlives model choice.
Why now
The leaderboard’s focus is timely for teams selecting coding-agent setups: it exposes comparable accuracy, cost, and repeatability measures for nine harnesses across two task suites, instead of treating a single model score as the whole system’s performance.
Why it matters
For teams building coding agents, the result is a reminder to evaluate the harness as part of the system, not as interchangeable packaging around a model. A team may trade a small score difference for lower cost, faster runs, or greater consistency. The leaderboard makes those trade-offs visible: Pi has a lower listed cost per solve than the overall leader, while the top five systems’ confidence intervals overlap the leader’s. That counsels against choosing by rank alone.
The published trajectories and pinned versions can help practitioners inspect how a system reached its result, but the benchmark is not a substitute for evaluation on a team’s own repositories, tools, permissions, and acceptance criteria. A useful adoption path is to shortlist a few harnesses, run the same representative tasks multiple times, and compare verified completion, cost, time, and failure modes. The value is less a universal winner than a reproducible basis for asking which harness works for a particular workload.
Limits and uncertainty
These are benchmark results, not independent evidence that the ranking transfers to production repositories or other task distributions. The board scores subsets of both benchmarks, and it notes task-specific environment and grader issues. Infrastructure failure retries are excluded, whereas agent errors and timeouts count as failures. Some trajectory checks are partial, including steps inside subagents and older trials with incomplete trajectories; the board says confirmed cheating is therefore a minimum. DeepSWE results also mix online and offline trials. The Space does not establish how the ranking changes with different models, providers, workloads, or harness versions.
Skill to build now
Build a small harness bake-off around tasks your team already reviews. Pin the model, provider, reasoning effort, task set, and harness versions; run each task more than once; and record pass rate, cost per accepted result, elapsed time, and consistency. Inspect trajectories for failure causes and shortcuts, then choose based on your own acceptance criteria rather than a single aggregate score.
Reader takeaway
A coding agent’s harness can change outcomes even when the model stays fixed, but leaderboard rank is not a deployment decision. Use the public comparison to form a shortlist, then test those harnesses repeatedly against your own work and measure cost alongside verified success.