Each app was deployed with Docker Compose and graded by browser agents that exercised its workflows. The study tested GPT 6 Sol and Claude Opus 5.5 at medium and maximum reasoning effort, with each of the eight configurations run once.
Teams cost 1.8 to 5.1 times as much as single agents. Only one of the four within-model comparisons showed a statistically significant score difference: GPT 6 Sol’s medium-effort team scored 84.9%, versus 77.6% for its single agent, a 7.3-point gain (p = 0.005). The other differences, ranging from −0.3 to +3.4 points, were not statistically significant. Costs and runtimes varied substantially: Claude Opus 5.5 at maximum effort cost a mean $122 per app as a team and took a median 174.1 minutes; its medium-effort single-agent setup cost $4.08 and took 20.3 minutes.
The results are specific to this benchmark and its implementation. They do not establish that teams generally fail or succeed; they show that model, reasoning effort, orchestration, and budget affect the trade-off. Vals AI notes that the work used one benchmark of full-stack web apps and may not carry over to other kinds of work.