{"id":"nr-agent-evaluation-is-shifting-to-harness-level-leaderboards-on-economic-8d6a2c","slug":"agent-evaluation-is-shifting-to-harness-level-leaderboards-on-economic-8d6a2c","type":"raw_signal","title":"Agent evaluation is shifting to harness-level leaderboards on economically valuable work","description":"Runta and Baseten's FrontierHarness Eval compares 22 harness configurations across terminal tasks, IBM and Kaggle's ScarfBench ranks agents on enterprise Java migration, SemiAnalysis's GDPx measures 206 long-horizon tasks across 11 high-value domains, and Joël Niklaus's leaderboard compares nine har","observed_at":"2026-11-19","record_date":"2026-11-19","date_kind":"observed_at","novelty":"notable","source_platform":"newruntime.com","topics":["codex"],"entities":["Runta","Baseten","FrontierHarness Eval","Harbor","Codex CLI","GLM-5.3","Kimi-K3","OpenCode","IBM","Kaggle"],"related_patterns":["harness-architecture-outlives-model-choice","verification-bandwidth-is-the-scarce-resource","agent-economics-moves-to-completed-work"],"source_url":"https://runta.com/blog/introducing-frontierharness-eval-v1-1","source_urls":["https://runta.com/blog/introducing-frontierharness-eval-v1-1","https://research.ibm.com/blog/ibm-and-kaggle-launch-leaderboard-for-java-framework-migration","https://newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities","https://huggingface.co/spaces/joelniklaus/harness-leaderboard"],"why_it_matters":"Runta and Baseten's FrontierHarness Eval compares 22 harness configurations across terminal tasks, IBM and Kaggle's ScarfBench ranks agents on enterprise Java migration, SemiAnalysis's GDPx measures 206 long-horizon tasks across 11 high-value domains, and Joël Niklaus's leaderboard compares nine harnesses with models fixed. The unit of evaluation is now the harness on real work, not the base model alone.","verification_level":"source-linked","signal_type":"synthesis","synthesis_of":[{"slug":"introducing-frontierharness-eval-v1-1-fd98d9","title":"Introducing FrontierHarness Eval v1.1","url":"https://newruntime.com/posts/introducing-frontierharness-eval-v1-1-fd98d9/"},{"slug":"ibm-and-kaggle-launch-java-framework-migration-leaderboard-ede4a9","title":"IBM and Kaggle launch Java framework migration leaderboard","url":"https://newruntime.com/posts/ibm-and-kaggle-launch-java-framework-migration-leaderboard-ede4a9/"},{"slug":"semianalysis-launches-gdpx-a-206-task-agentic-benchmark-30b569","title":"SemiAnalysis launches GDPx, a 206-task agentic benchmark across 11 high-value domains; top model scores 12%","url":"https://newruntime.com/posts/semianalysis-launches-gdpx-a-206-task-agentic-benchmark-30b569/"},{"slug":"joel-niklaus-releases-harness-leaderboard-comparing-nine-coding-f0efa5","title":"Joël Niklaus releases Harness Leaderboard comparing nine coding-agent harnesses on open models","url":"https://newruntime.com/posts/joel-niklaus-releases-harness-leaderboard-comparing-nine-coding-f0efa5/"}],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"signal:agent-evaluation-is-shifting-to-harness-level-leaderboards-on-economic-8d6a2c","retrieval_nugget":"Runta and Baseten's FrontierHarness Eval compares 22 harness configurations across terminal tasks, IBM and Kaggle's ScarfBench ranks agents on enterprise Java migration, SemiAnalysis's GDPx measures 206 long-horizon tasks across 11 high-value domains, and Joël Niklaus's leaderboard compares nine harnesses with models fixed.","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/signals/agent-evaluation-is-shifting-to-harness-level-leaderboards-on-economic-8d6a2c/","markdown":"https://newruntime.com/signals/agent-evaluation-is-shifting-to-harness-level-leaderboards-on-economic-8d6a2c.md","json":"https://newruntime.com/signals/agent-evaluation-is-shifting-to-harness-level-leaderboards-on-economic-8d6a2c.json"},"markdown_url":"https://newruntime.com/signals/agent-evaluation-is-shifting-to-harness-level-leaderboards-on-economic-8d6a2c.md","next_reads":[{"type":"topic","path":"/topics/codex/","url":"https://newruntime.com/topics/codex/","title":"Codex","reason":"Explore the codex topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/codex.md","json_url":"https://newruntime.com/topics/codex.json"},{"type":"collection","path":"/signals/","url":"https://newruntime.com/signals/","title":"Signals","reason":"Open the signals and the raw-signal archive.","media_type":"text/html","json_url":"https://newruntime.com/signals.json"},{"type":"collection","path":"/sources/","url":"https://newruntime.com/sources/","title":"Sources","reason":"See the original sources this archive cites.","media_type":"text/html"}]}
