{"type":"analysis","id":"analysis:research-agent-evals-are-moving-to-workflow-specific-benchmarks-and-the-vendors-who","slug":"research-agent-evals-are-moving-to-workflow-specific-benchmarks-and-the-vendors-who","title":"Agent evals are moving to workflow-specific benchmarks, and the vendors who build them usually top them","description":"Between June and September 2026, Firecrawl, GitHub, LlamaIndex, Warp and OpenAI each published evals that measure one specific job: retrieval, orchestration, extraction, task bake-offs or misalignment in deployment. They are useful as methods. As leaderboards they need care, because in most of them the publisher scores itself.","published_at":"2026-10-24T15:00:00.000Z","updated_at":"2026-10-24T15:00:00.000Z","record_date":"2026-10-24","topics":["evals","github","search","research"],"source_urls":["https://www.firecrawl.dev/benchmarks/devdex","https://github.blog/ai-and-ml/github-copilot/project-hydrafusion-frontier-quality-via-multi-model-orchestration/","https://venturebeat.com/orchestration/githubs-hydrafusion-cuts-ai-coding-costs-in-every-benchmark-it-only-matches-quality-in-one","https://llamaindex.ai/blog/introducing-extractbench","https://warp.dev/blog/warp-factory-benchmarks","https://alignment.openai.com/validating-public-evals"],"language":"en","routes":{"html":"https://newruntime.com/analysis/research-agent-evals-are-moving-to-workflow-specific-benchmarks-and-the-vendors-who/","markdown":"https://newruntime.com/analysis/research-agent-evals-are-moving-to-workflow-specific-benchmarks-and-the-vendors-who.md","json":"https://newruntime.com/analysis/research-agent-evals-are-moving-to-workflow-specific-benchmarks-and-the-vendors-who.json"},"author":{"role":"author","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"editorial_provenance":{"desk":"authorial","target_ref":"deep_research:cc0bf767-08e3-4b42-be18-cbc61966be36","authorial_article_id":"cc0bf767-08e3-4b42-be18-cbc61966be36"},"next_reads":[{"type":"topic","path":"/topics/evals/","url":"https://newruntime.com/topics/evals/","title":"Evals","reason":"Explore the evals topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/evals.md","json_url":"https://newruntime.com/topics/evals.json"},{"type":"topic","path":"/topics/github/","url":"https://newruntime.com/topics/github/","title":"Github","reason":"Explore the github topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/github.md","json_url":"https://newruntime.com/topics/github.json"},{"type":"topic","path":"/topics/search/","url":"https://newruntime.com/topics/search/","title":"Search","reason":"Explore the search topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/search.md","json_url":"https://newruntime.com/topics/search.json"},{"type":"collection","path":"/analysis/","url":"https://newruntime.com/analysis/","title":"Analysis","reason":"Open the latest authorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/analysis.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}