{"id":"nr-agents-are-being-measured-on-real-economic-work-and-they-still-fail-mo-b60b07","slug":"agents-are-being-measured-on-real-economic-work-and-they-still-fail-mo-b60b07","type":"raw_signal","title":"Agents are being measured on real economic work, and they still fail most of it","description":"New benchmarks now measure agents on long, economically valuable tasks.","observed_at":"2026-11-03","record_date":"2026-11-03","date_kind":"observed_at","novelty":"notable","source_platform":"newruntime.com","topics":["claude","claude-code","codex"],"entities":["SemiAnalysis","GDPx","Claude Opus 5.5","Gemini 4 Carbon","GPT-6 Astra","DeepSeek 4.1 Flash","Claude Code","Codex","Xplore","RWE-bench"],"related_patterns":["agent-economics-moves-to-completed-work","scientific-agents-need-artifact-evidence","ai-engineering-evaluations-are-splitting-into-task-specific"],"source_url":"https://newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities","source_urls":["https://newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities","https://gdpx.semianalysis.com/","https://xplore.ai/rwe-bench","https://openai.com/index/lifeval","https://arxiv.org/abs/2610.37739","https://github.com/aliyun/ai-agent-handbook/blob/main/2026-agent-survey-report.md"],"why_it_matters":"New benchmarks now measure agents on long, economically valuable tasks. They cover specialist domains, real engineering, everyday personal-assistant work and open math problems. The results are weak: the best model scores 12% across 206 domain tasks, and agents without web access rediscovered at most 7 of 40 solved problems. A developer survey found that only 18% of teams reach production and 10% reach 90% success, so builders should plan for partial completion.","verification_level":"source-linked","signal_type":"synthesis","synthesis_of":[{"slug":"semianalysis-launches-gdpx-a-206-task-agentic-benchmark-30b569","title":"SemiAnalysis launches GDPx, a 206-task agentic benchmark across 11 high-value domains; top model scores 12%","url":"https://newruntime.com/posts/semianalysis-launches-gdpx-a-206-task-agentic-benchmark-30b569/"},{"slug":"semianalysis-releases-gdpx-an-agentic-benchmark-across-11-56f9ba","title":"SemiAnalysis releases GDPx, an agentic benchmark across 11 high-value domains where the best model scores 12%","url":"https://newruntime.com/posts/semianalysis-releases-gdpx-an-agentic-benchmark-across-11-56f9ba/"},{"slug":"rwe-bench-real-world-engineering-bench-195f0e","title":"RWE-bench: Real World Engineering Bench","url":"https://newruntime.com/posts/rwe-bench-real-world-engineering-bench-195f0e/"},{"slug":"openai-unveiled-lifeval-a-benchmark-grading-models-on-44478e","title":"OpenAI unveiled LifeVal, a benchmark grading models on everyday personal-assistant tasks.","url":"https://newruntime.com/posts/openai-unveiled-lifeval-a-benchmark-grading-models-on-44478e/"},{"slug":"openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554","title":"OpenFrontiers benchmark finds LLM agents rarely rediscover recently solved math problems without web access","url":"https://newruntime.com/posts/openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554/"},{"slug":"includes-a-2026-developer-survey-only-18-of-8ae18f","title":"Includes a 2026 developer survey: only 18% of teams reach production, 10% hit 90% success.","url":"https://newruntime.com/posts/includes-a-2026-developer-survey-only-18-of-8ae18f/"}],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"signal:agents-are-being-measured-on-real-economic-work-and-they-still-fail-mo-b60b07","retrieval_nugget":"New benchmarks now measure agents on long, economically valuable tasks. They cover specialist domains, real engineering, everyday personal-assistant work and open math problems. The results are weak: the best model scores 12% across 206 domain tasks, and agents without web access rediscovered at most 7 of 40 solved problems.","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/signals/agents-are-being-measured-on-real-economic-work-and-they-still-fail-mo-b60b07/","markdown":"https://newruntime.com/signals/agents-are-being-measured-on-real-economic-work-and-they-still-fail-mo-b60b07.md","json":"https://newruntime.com/signals/agents-are-being-measured-on-real-economic-work-and-they-still-fail-mo-b60b07.json"},"updated_at":"2026-11-07","next_reads":[{"type":"topic","path":"/topics/claude/","url":"https://newruntime.com/topics/claude/","title":"Claude","reason":"Explore the claude topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/claude.md","json_url":"https://newruntime.com/topics/claude.json"},{"type":"topic","path":"/topics/claude-code/","url":"https://newruntime.com/topics/claude-code/","title":"Claude Code","reason":"Explore the claude-code topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/claude-code.md","json_url":"https://newruntime.com/topics/claude-code.json"},{"type":"topic","path":"/topics/codex/","url":"https://newruntime.com/topics/codex/","title":"Codex","reason":"Explore the codex topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/codex.md","json_url":"https://newruntime.com/topics/codex.json"},{"type":"collection","path":"/signals/","url":"https://newruntime.com/signals/","title":"Signals","reason":"Open the signals and the raw-signal archive.","media_type":"text/html","json_url":"https://newruntime.com/signals.json"},{"type":"collection","path":"/sources/","url":"https://newruntime.com/sources/","title":"Sources","reason":"See the original sources this archive cites.","media_type":"text/html"}]}
