---
type: "post"
slug: "agent-evaluation-is-moving-from-leaderboards-to-operating-diagnostics"
title: "Agent evaluation is moving from leaderboards to operating diagnostics"
description: "Open tasks, stack benchmarks, delivery checks, and behavior catalogs expose why an agent succeeded or failed."
retrieval_nugget: "Open tasks, stack benchmarks, delivery checks, and behavior catalogs expose why an agent succeeded or failed."
published_at: "2026-08-17"
updated_at: "2026-08-17"
record_date: "2026-08-17"
date_kind: "scheduled_at"
topics: ["agent-evals","observability","harnesses","governance"]
entities: ["mercor.com","dora.dev","agentbehavior.dev"]
editorial_format: "pattern_review"
basket_id: "64af3bcb-1c2d-42a9-a664-91510a61d75a"
basket_revision: 1
source_urls: ["https://mercor.com/apex/oss-benchmarks/oss-terminal-bench-2-1-leaderboard/sample-task","https://dora.dev/insights/quickcheck-updates","https://www.agentbehavior.dev/"]
visual_decision: "text_only"
recovery_incident: "NR-2026-08-15-HERMES-SITE-COPY"
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:agent-evaluation-is-moving-from-leaderboards-to-operating-diagnostics"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:b2f8378508755f4a6d06535354c18ae9900c5c5c702a56213563a30fb2c76ce1","reviewed_at":"2026-08-15T20:30:00.000Z","source_evidence_count":3,"verified_claim_count":2}
routes: {"html":"https://newruntime.com/posts/agent-evaluation-is-moving-from-leaderboards-to-operating-diagnostics/","markdown":"https://newruntime.com/posts/agent-evaluation-is-moving-from-leaderboards-to-operating-diagnostics.md","json":"https://newruntime.com/posts/agent-evaluation-is-moving-from-leaderboards-to-operating-diagnostics.json"}
---

# Agent evaluation is moving from leaderboards to operating diagnostics

## Retrieval answer

Open tasks, stack benchmarks, delivery checks, and behavior catalogs expose why an agent succeeded or failed.

Agent evaluation is acquiring diagnostic surfaces around the score. Mercor publishes sample tasks for Terminal-Bench and related benchmarks, DORA's Quick Check connects delivery practices to concrete improvement areas, and AgentBehavior organizes observed behaviors that can appear during an agent trajectory.

The shared move is from “which model ranks first?” to “which system configuration produced this result?” A meaningful record includes the model version, harness, tools, context budget, cost, reviewer interventions, and any unsafe actions along the way. Without that data, a leaderboard cannot explain whether the gain will survive a different repository or policy.

For production selection, eval output should lead to an operating decision: allow, restrict, route, review, or reject a model-plus-harness combination. Open tasks and behavior traces make that decision auditable. They also make regressions easier to localize when a model, tool, or prompt changes independently.

## Sources

- [Mercor sample task](https://mercor.com/apex/oss-benchmarks/oss-terminal-bench-2-1-leaderboard/sample-task)
- [DORA Quick Check](https://dora.dev/insights/quickcheck-updates)
- [AgentBehavior](https://www.agentbehavior.dev/)