Coding agents finish long tasks when someone else decides when they're done

What we checked

• Does the cluster show a broader architecture shift that a single release note can't capture? partly confirmed. Three separate sources point the same way: tell agents what 'done' means through something outside the agent that checks the work. These are Factory's three-role system with a hidden validator, the SpecFirst paper's spec-first, two-agent pipeline, and Warp's eval tool built from a team's own task history. Factory's gains come from single runs that weren't compute-matched, so this is a well-supported pattern rather than an established result. [1][4][5]
• (Added) Is 'Factory Benchmarks' a product of Factory, the company behind Droid? refuted. Benchmarks is an early-access feature of Warp Factories, made by Warp. The X post in the cluster comes from Warp's account. Factory's own post is about its Missions product and research on ProgramBench. [5][6][1]
• (Added) Does Warp's tool mirror each run's environment, secrets and MCPs, as the social post says? unresolved. Neither Warp's launch blog nor its Benchmarks product page mentions secrets or MCPs. Both say past runs can be replayed 'from their original state'. The X post itself returned HTTP 402 and could not be read. [6][5]

What we found

• Factory took 24 hard tasks from ProgramBench, chosen because public best scores on them were low, and compared two setups. One was a single agent that decides for itself when it's done. The other was a system with three roles: an orchestrator, an implementer, and a validator that builds a weighted, byte-for-byte differential test suite before any implementation begins. The implementer never sees that suite or its raw results; Factory calls this information barrier 'the wall'. Median hidden-suite scores rose for all three models Factory names: 'Fable 5' from 56.7 to 89.3, 'Kimi K3' from 45.1 to 75.4, and 'GPT-5.6 Sol' from 48.6 to 66.2. [1]
• GDAL, a geospatial CLI with about 600K reachable lines, went from 35.8% to 90.3% with Fable, and the codebase grew from 17K to 115K lines. Factory also reports 7-Zip rising from 54% to 95% and DuckDB from 34% to 80%. The cost was large: about 14 times the credits and 196.9 hours versus 15.0 hours of wall-clock time on GDAL. Factory says budget was not the limiting factor; single agents simply chose to stop. [1]
• Factory lists its own caveats. Each cell is a single run with no variance estimate, and runs were not compute-matched. Six Fable cells used Opus instead after Fable was safety-blocked. Three of 144 cells were interrupted, some evaluations timed out so their scores are floors, and a few system runs scored lower, such as Sol on lnav and duckdb. Factory says it is building this structure into the next generation of Missions, with a waitlist. [1]
• ProgramBench (Yang et al., arXiv 2605.03546, May 2026) gives agents a program and its documentation and grades their rebuilds with hidden behavioral tests generated by fuzzing. It has 200 tasks. In the paper, none of nine models fully solved any task. As of October 12, 2026, Vals.ai's leaderboard shows Claude Opus 5.5 at 18.50% fully resolved and an 87.0% raw pass rate. Factory's scores use the hidden-suite metric, so compare them with raw pass rates, not with full-resolution rates. [2][3]
• SpecFirst (arXiv 2607.27167, July 2026) is an independent academic study of a related idea. A spec agent probes the program and writes a structured specification, and a separate agent writes code from it. Across all 200 ProgramBench tasks and four models, it raised test pass rates by 6.9% to 21.3% over a single-loop baseline, and the authors report the gains as statistically significant. Its gains are smaller than Factory's, but it supports the idea that separating 'what done means' from implementation helps. [4]
• Warp's Benchmarks, in early access since September 3, 2026, builds task sets from a team's past Factory runs and replays them from their original state. It varies the model and the harness (support for Claude Code and Codex harnesses is planned), and scores runs with LLM-as-a-judge rubrics. Results appear as cost-versus-quality Pareto charts with a recommendation you can apply in one click. Warp says it used its internal suite, WarpBench, to cut internal cost per PR by about 63%. It then routes simple frontend tasks to Grok 4.6 (med), while GPT 5.6 Sol (high) led overall. Its pages don't mention mirroring secrets or MCPs. [5][6]

What it means

For long tasks, the evidence supports giving 'is it done?' to something other than the agent that wrote the code. That can be a validator agent that sees the reference behavior but keeps its tests hidden from the implementer, as in Factory's setup, or a spec written before any code, as in SpecFirst. Expect this to cost an order of magnitude more time and compute, because the system keeps working where a single agent would have stopped. Budget for that. Also add safeguards a single run can't show: repeat runs, timeouts, and checks for cases where the multi-agent system scores worse than the single agent, which Factory itself observed.

When choosing models, Warp's approach is the practical counterpart: score candidate models and configurations on replays of your own past tasks, against your own rubric, with repeated runs. Public leaderboards are less useful here. Keep the two Factory items separate when sourcing: Factory (Droid, Missions) published the research, and Warp built the benchmark tool. Treat vendor numbers from both as directional. Factory's numbers are single runs, and Warp's cost savings are internal figures with LLM-judged quality.

Still open

• Do Factory's gains hold up with repeated runs and with compute-matched single-agent baselines given the same budget?
• When will the validator design actually ship in Factory's Missions product, and in what form?
• Does Warp's replay recreate secrets, MCP servers and the full environment, as the social post claims? Its public pages don't say.
• How reliable are LLM-as-a-judge scores in Warp's tool compared with executable tests?