---
type: analysis
slug: research-agent-evals-are-moving-to-workflow-specific-benchmarks-and-the-vendors-who
title: "Agent evals are moving to workflow-specific benchmarks, and the vendors who build them usually top them"
description: "Between June and September 2026, Firecrawl, GitHub, LlamaIndex, Warp and OpenAI each published evals that measure one specific job: retrieval, orchestration, extraction, task bake-offs or misalignment in deployment. They are useful as methods. As leaderboards they need care, because in most of them the publisher scores itself."
published_at: 2026-10-24T15:00:00.000Z
topics: ["evals","github","search","research"]
source_urls: ["https://www.firecrawl.dev/benchmarks/devdex","https://github.blog/ai-and-ml/github-copilot/project-hydrafusion-frontier-quality-via-multi-model-orchestration/","https://venturebeat.com/orchestration/githubs-hydrafusion-cuts-ai-coding-costs-in-every-benchmark-it-only-matches-quality-in-one","https://llamaindex.ai/blog/introducing-extractbench","https://warp.dev/blog/warp-factory-benchmarks","https://alignment.openai.com/validating-public-evals"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
The newsroom's cluster combines several kinds of evals: search-tool evals (Firecrawl's DevDex), orchestration claims on coding benchmarks (GitHub's Project HydraFusion on TerminalBench 2.1), document extraction (LlamaIndex ExtractBench), benches built from a team's own tasks (Warp Factories) and safety evals (OpenAI's study of whether public WildChat data predicts misalignment in production). What they have in common is the unit of measurement. None of them asks how good a model is in general. Each asks how well a specific component or configuration does a specific job, such as finding the issue that fixes a bug, extracting every row of a 50-page document, or finishing a coding task at a given cost.

For builders, this matters because these benchmarks are starting to drive purchasing and routing decisions: which search API an agent gets, whether to put a router in front of a frontier model, which extraction vendor to use. Most of the published numbers come from the companies selling the winning product. So the useful question is how much of each eval's method you can reuse, and how much of each leaderboard you should discount. Two items in the cluster were not evals at all. The Brex "benchmark" page covers customer spending data, and eval-ai.com is a course landing page.

## What we checked

- **The current eval wave is converging on workflow-specific measurement because generic model scores no longer predict deployable agent utility.** partly confirmed. The convergence on workflow-specific measurement is clear. DevDex scores developer retrieval, ExtractBench scores document extraction, Warp builds benches from a team's own coding tasks, and GitHub uses CheckpointBench, which replays real Copilot sessions. The causal claim has only partial support: GitHub calls TerminalBench 2.1 "relatively saturated," ExtractBench finds little separation between systems on short documents, and OpenAI finds public chat data is a weak proxy for agentic risk (about 37x error). None of these sources directly shows that generic scores fail to predict deployed utility. [1][2][4][5][6]
- **(Added) Are the headline results in this wave mostly vendor-run, with the publisher ranking first?** confirmed. Firecrawl's Developer Index leads Firecrawl's DevDex, and LlamaIndex's Agentic Plus leads LlamaIndex's ExtractBench. GitHub's HydraFusion numbers come from GitHub's own offline tests, with routing policies tuned across the same evaluation sets, and Warp reports gains on its internal data. Firecrawl and LlamaIndex do publish their data and harnesses so others can reproduce the results. [1][4][2][3]

## What we found

- DevDex (v1.0.0, Aug 21, 2026) scores 1,179 developer queries across three tracks: repository, issue/PR and documentation. Each search tool is driven by Claude Opus 4.8 with 10 results per call, scoring uses deterministic Recall@10 and MRR@10 against gold URLs with no model judge, and a memorization check drops any query the model can answer without search. Firecrawl's Developer Index leads overall at 63.1%, ahead of Parallel (57.7%) and Exa (53.7%). On the docs track the best score is 47.2%. Native web search scores 80.7% on repositories but only 27.5% on issues/PRs. [1]
- The DevDex track breakdown shows why a single blended score misleads. Parallel wins repositories (81.9%), Context7 nearly matches the leader on docs (46.6%) but scores 0.8% on repositories, and the overall ranking depends on weighting the three tracks equally. A 594-item public sample and the harness are on GitHub. The remaining items are held back. [1]
- GitHub's HydraFusion (Sept 4, 2026) routes each coding task through one of three patterns, called Single, Cascade or Critique, across several models. Against Claude Opus 5 it reports +4.9 points of verified quality at 67% lower estimated cost on TerminalBench 2.1, −1.5 points at 36% lower cost on DeepSWE, and −0.1 points at 65% lower cost on its internal CheckpointBench. GitHub says the policies were tuned across the evaluation sets, that TerminalBench 2.1 is "relatively saturated," and that the preview targets first-turn, single-prompt tasks. [2]
- VentureBeat points out that HydraFusion cut cost in all three tests but matched or beat Opus 5 on quality in only one, and argues the same pattern runs across the routing market, including Nvidia's NeMo Switchyard and OpenRouter's Auto router. Read this way, routing evals measure cost-quality trade-offs, not quality gains. [3]
- ExtractBench (Aug 11, 2026) covers 370 documents (4,869 pages) in 67 document types and reports results on several axes, including length and perception (scans, handwriting). LlamaIndex reports that its own Agentic Plus tier ranks first at 95.6% value F1 and 8.1¢ per page, that commercial vision-language models fall below 35% recall on documents over 50 pages, and that short documents barely separate systems, leaving price as the main differentiator there. [4]
- Warp's Factories Benchmarks (Sept 3, 2026) let teams build a bench from their own past agent runs. The bench runs a tasks × configurations matrix, scores each run with an LLM judge against rubrics the team defines, and plots the results as Pareto charts. Warp reports a 63% cost reduction on certain task types from routing simple frontend tasks to a cheaper model. It gives no sample sizes or variance figures. [5]
- OpenAI's alignment team (June 16, 2026) found that regenerating responses to roughly 100k WildChat prompts predicted production misalignment rates within about 1.04 orders of magnitude in 95% of cases for ordinary chat. For six agentic categories the error was about 37x. The authors call for public datasets that cover long-context, tool-using, multi-session work. [6]

## What it means

Take the methods and discount the leaderboards. The best design choices here carry over to other teams. DevDex scores deterministically against gold URLs and removes anything the model already knows from memory. ExtractBench breaks results out by document length and scan quality, which is where systems actually differ. GitHub counts the cost of every step in the workflow, and both GitHub and Warp build benches from replayed real sessions. A small internal suite that combines these ideas is worth more than any vendor's ranking: frozen tasks taken from your own traffic, deterministic checks where possible, results reported per track and not blended, and cost per verified success. When you read a vendor-run result, check three things. Did the publisher tune on the same evaluation set? Is the benchmark saturated? Does the headline number come from the one favorable track or test?

On routers in particular, the evidence supports large cost reductions with roughly flat or slightly lower quality. It does not support quality gains. Before rolling one out, run it on your own multi-turn tasks, since HydraFusion's preview does not claim multi-turn coverage. On safety, OpenAI's result suggests that public chat proxies are acceptable for ordinary chat failures but not for agentic risk, so tool-using behavior needs evals built from real agent trajectories.

## Still open

- No independent party has yet reproduced DevDex or ExtractBench rankings using the public samples and harnesses.
- GitHub has not published HydraFusion results on multi-turn, long-running repository work, or against a GPT-5.6 Sol baseline with numbers.
- It is unclear how much the DevDex results depend on the single harness model (Claude Opus 4.8) and on default tool configurations.
- Warp's 63% cost figure has no published sample sizes, variance figures or quality deltas.

## Sources

1. [Firecrawl – Developer Retrieval Benchmark (DevDex)](https://www.firecrawl.dev/benchmarks/devdex) (primary)
2. [GitHub Blog – Project HydraFusion: Frontier quality via multi-model orchestration](https://github.blog/ai-and-ml/github-copilot/project-hydrafusion-frontier-quality-via-multi-model-orchestration/) (primary)
3. [VentureBeat – GitHub's HydraFusion cuts AI coding costs in every benchmark. It only matches quality in one.](https://venturebeat.com/orchestration/githubs-hydrafusion-cuts-ai-coding-costs-in-every-benchmark-it-only-matches-quality-in-one)
4. [LlamaIndex – Introducing ExtractBench](https://llamaindex.ai/blog/introducing-extractbench) (primary)
5. [Warp – Benchmarks in Warp Factories](https://warp.dev/blog/warp-factory-benchmarks) (primary)
6. [OpenAI Alignment – Can public chat data predict real-world AI misalignments?](https://alignment.openai.com/validating-public-evals) (primary)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Github](https://newruntime.com/topics/github.md): Explore the github topic hub.
- topic hub: [Search](https://newruntime.com/topics/search.md): Explore the search topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)