AnalysisNov 10, 20267 min read9 sources

AgenC's $27 Terminal-Bench 2.1 run checks out on Harbor, but it is a self-run result, not a board entry

In one minute

The public Harbor job matches AgenC's raw score and cost; the 84.5% headline rests on AgenC's own contamination flags in a custom environment, and DeepSeek's card shows the model scores similarly in other harnesses.

AgenC, an MIT-licensed terminal coding agent from Tetsuo.AI, published a page claiming its agent with DeepSeek V4.1 Flash scored 84.5% on Terminal-Bench 2.1 for $27, while every Harbor Hub leaderboard entry cost at least $134. The claim matters because it suggests a cheap open model plus a cache-friendly harness can get near frontier coding-agent scores at a small fraction of the cost.

The newsroom flagged it because the result is self-published. We checked the public Harbor Hub job, the Terminal-Bench 2.1 submission rules, official task timeouts, DeepSeek's model card and pricing, and AgenC's earlier claims.

What we checked

  • Can the public trajectories, contamination flags, costs, and Terminal-Bench scoring be independently reproduced and audited? partly confirmed. The Harbor job is public with all 445 trials, a 0.86 average reward that matches AgenC's pre-flag 85.6%, and a metered $27.21 cost. The five contamination flags are AgenC's own judgment, the environment is custom, and no maintainer or third-party audit was found. [3][1][4]
  • Is the 84.5% result an official Terminal-Bench 2.1 leaderboard entry? refuted. AgenC itself says it is a public run, not a board entry, and the Terminal-Bench 2.1 repo says only maintainer-run submissions are being added. [2][4]
  • Does the AgenC harness itself drive the high score with DeepSeek V4.1 Flash? refuted. DeepSeek reports 84.1 to 90.6 on Terminal-Bench 2.1 for the same model across several harnesses, so the score is in line with the model; the standout is cost. [6]

What we found

  • AgenC's benchmark page says AgenC Core 0.19.24 with DeepSeek V4.1 Flash at max effort scored 84.5% on Terminal-Bench 2.1 for $27, and ranks it against Harbor Hub board entries. Its DeepSeek page adds that this is 376 of 445 trials and that it is a public run on Harbor Hub, not a board entry. [1][2]
  • The linked Harbor Hub job is public, shows all 445 trials finished (442 completed, 3 errors, 0 timeouts) and an average reward of 0.86, which matches AgenC's pre-flag 85.6% (381 of 445) rather than the headline 84.5%. Harbor lists $27.21 inference cost, 1.23 billion total tokens and a 97% cache hit rate, so the $27 figure is Harbor's own metering of the same run. [3][1]
  • The drop from 85.6% to 84.5% comes from AgenC's own contamination rule: trials where the agent saw a task's solution or test file were counted as failures, five in total. The DeepSeek page also says trials where the agent looked up an answer online were failed, which implies the agent had some network access. The job config served on the Harbor page uses a custom filtered Docker environment with a host package cache and read-only mounts for the AgenC runtime and a helper, not a stock Harbor sandbox. We did not find a public per-trial list of the five flagged trials. [1][2][3]
  • The Terminal-Bench 2.1 repository says community submissions are closed and only maintainer-run results join the leaderboard; submissions need at least five public trials per task. AgenC meets the five-trial and public-upload bar, but the result has not gone through maintainer review, so comparing it with board rows mixes audited and self-run results. For sampled tasks the run's agent timeouts match the official task files (for example 750 seconds on overfull-hbox). [4][5][3]
  • DeepSeek's own model card reports 90.6% Pass@1 on Terminal-Bench 2.1 for V4.1 Flash using its Minimal harness, and between 84.1 and 90.3 across Claude Code, Codex, OpenCode, Pi and mini-SWE with three samples per task. That places AgenC's 84.5% inside the range other scaffolds reach with the same model, so the score mostly reflects the model; the distinctive claim is cost. [6]
  • AgenC's own news page still lists the earlier 0.19-series result of 76.2% for $34 on Terminal-Bench 2.1, posted October 16. The new run is about eight points higher two weeks later, after AgenC said it used official timeouts and resources. A gain of that size between agent point releases is plausible but is exactly what an independent rerun should confirm. [7][1]
  • Low cost depends on DeepSeek's pricing: Flash cache-hit input is listed at $0.003 to $0.006 per million tokens, and off-peak rates are half of peak. With 97% cache hits on 1.23 billion tokens, a run at another time of day or with a less cache-friendly prompt would cost more, so $27 is a property of this run and price schedule, not a fixed number. [8][3]

What it means

For builders, the cost result is the useful part: a run with 97% cache hits on DeepSeek Flash pricing really did cost about $27 for 445 trials, as metered by Harbor rather than by AgenC. If you run long agent loops on DeepSeek, keeping the prompt prefix stable for caching matters more than which scaffold you choose. Treat the 84.5% as the model's capability in a reasonable harness, not as proof the harness beats Claude Code or Codex, because those board rows use different models and went through maintainer review.

Before relying on it, rerun a slice of tasks yourself with the stock Harbor environment and network rules, and compare cost at your own time of day. The comparison that would settle the harness question is the same model across harnesses under maintainer conditions.

Competing readings

  • The result is genuine and reproducible. Its real contribution is low cost from a cache-friendly harness on a cheap model, with a score typical of that model (likely, about 60%). What would tell it apart: Harbor's metered reward and cost match AgenC's raw score and $27 figure [3][1]; DeepSeek evaluates the same model in its own harness and in a scaffold comparison, which points to similar scores in other harnesses [6].
  • The score is inflated by a non-standard environment and contamination flagging AgenC applied itself, and it would fall noticeably under maintainer-run conditions (unlikely, about 28%). What would tell it apart: The contamination flags and the environment were the submitter's own, with possible network access [1][2][3]; The run is not a maintainer-reviewed leaderboard entry [2][4].
  • The AgenC harness itself is what lifts DeepSeek Flash to a near-frontier score (very unlikely, about 12%). What would tell it apart: AgenC reports 84.5% with this model in its own harness [1]; DeepSeek benchmarks the same model in its own Minimal harness and in a scaffold comparison, so the harness effect can be checked against those numbers [6].

Base rate

The New Runtime archive holds only 4 earlier research stories about companies, models, organisations, products or standards from the last 12 months that are at least 3 months old: too few (minimum 5) for a base rate. The archive holds only 4 comparable earlier research stories, of which 3 were followed by a release. That is below the minimum of 5, so it is too few to lean on, and the conclusion rests on the primary evidence alone.

What would change this view

  • A maintainer-run Terminal-Bench 2.1 entry for AgenC Core with DeepSeek V4.1 Flash scoring well below 84.5% (signal): The custom environment or self-flagging inflated the published score.
  • DeepSeek raises Flash cache-hit or output prices (pricing signal): The $27 cost advantage would shrink regardless of the harness.
  • An independent rerun of the public trajectories finds more than five trials that read solution or test files (signal): AgenC's contamination handling was incomplete and the headline overstates the score.

Still open

  • Which five trials were flagged for seeing solution or test files, and would maintainers flag the same ones?
  • Does the score hold in a stock Harbor sandbox without the custom filtered environment and package cache?
  • Will Terminal-Bench maintainers rerun or list AgenC Core once submissions reopen?
  • How much of the jump from 76.2% to 84.5% comes from agent changes versus run setup?

Evidence / sources

  1. 1get.agenc.ag/terminal-bench
  2. 2get.agenc.ag/deepseek
  3. 3hub.harborframework.com/jobs/2bf2c84d-83c2-47a3-9341-e635c45bd329
  4. 4github.com/harbor-framework/terminal-bench-2-1
  5. 5raw.githubusercontent.com/harbor-framework/terminal-bench-2-1/main/tasks/overfull-hbox/task.toml
  6. 6huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash
  7. 7agenc.ag/news
  8. 8api-docs.deepseek.com/quick_start/pricing
  9. 9github.com/tetsuo-ai/agenc-core
All Analysis

Next reads

Read next

Open the graph
  1. 01topicClaudeExplore the claude topic hub.
  2. 02topicClaude CodeExplore the claude_code topic hub.
  3. 03topicCodexExplore the codex topic hub.
  4. 04archiveAnalysisOpen the latest authorial analysis.
  5. 05sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract