SignalNov 19, 20261 min read4 sources

Agent evaluation is shifting to harness-level leaderboards on economically valuable work

In one minute

Runta and Baseten's FrontierHarness Eval compares 22 harness configurations across terminal tasks, IBM and Kaggle's ScarfBench ranks agents on enterprise Java migration, SemiAnalysis's GDPx measures 206 long-horizon tasks across 11 high-value domains, and Joël Niklaus's leaderboard compares nine harnesses with models fixed.

The unit of evaluation is now the harness on real work, not the base model alone.

From the Field Notes:

Evidence / sources

  1. 1runta.com/blog/introducing-frontierharness-eval-v1-1
  2. 2research.ibm.com/blog/ibm-and-kaggle-launch-leaderboard-for-java-framework-migration
  3. 3newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities
  4. 4huggingface.co/spaces/joelniklaus/harness-leaderboard
All Signals

Next reads

Read next

Open the graph
  1. 01topicCodexExplore the codex topic hub.
  2. 02archiveSignalsOpen the signals and the raw-signal archive.
  3. 03sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract