---
type: "post"
id: "do-agent-teams-pay-off-a-case-study-04444a"
slug: "do-agent-teams-pay-off-a-case-study-04444a"
title: "Do Agent Teams Pay Off? A Case Study on Vibe Code Bench (18 minute read)"
description: "**Thesis:** Agent teams should be judged by verified task outcomes relative to their added cost and time, not by parallelism alone."
retrieval_nugget: "**Thesis:** Agent teams should be judged by verified task outcomes relative to their added cost and time, not by parallelism alone."
published_at: "2026-10-12"
updated_at: "2026-10-12"
record_date: "2026-10-12"
date_kind: "published_at"
topics: []
source_urls: ["https://www.vals.ai/blogs/multi-agent-vibe-code-bench"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:do-agent-teams-pay-off-a-case-study-04444a"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:c4807bdee8b4c25a24c37397389cfbd5dabd3d122ec00f89d69d9019c9d9160a","reviewed_at":"2026-10-12T14:06:23.017Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Agent teams should be judged by verified task outcomes relative to their added cost and time, not by parallelism alone.","observed_facts":[{"text":"Vals AI tested GPT 6 Sol and Claude Opus 5.5 on 50 Vibe Code Bench apps at medium and maximum reasoning effort; each of the eight configurations ran once.","source_urls":["https://www.vals.ai/blogs/multi-agent-vibe-code-bench"]},{"text":"Teams cost 1.8 to 5.1 times as much as single agents; only GPT 6 Sol at medium effort had a statistically significant score gain, 7.3 points (p = 0.005).","source_urls":["https://www.vals.ai/blogs/multi-agent-vibe-code-bench"]},{"text":"The Vals AI report says Sol generally split work early and tested, while Opus often worked in sequential waves.","source_urls":["https://www.vals.ai/blogs/multi-agent-vibe-code-bench"]}],"mechanism":"Delegation creates additional working contexts and orchestration overhead; repeated cached input for subagents drove much of the extra spend, while orchestration style affected whether parallelism reduced runtime.","why_now":"Both OpenAI and Anthropic now offer native multi-agent support, making the operational choice between delegation and a stronger single-agent run immediate for teams building software with these systems.","implications":["Benchmark delegation against higher reasoning effort on representative tasks, tracking score, cost, and runtime together.","Do not treat a team’s peak score as a default choice without accounting for its cost and time."],"evidence_boundary":"The study covers one benchmark of full-stack web apps, 50 tasks, and one run per configuration; results may not transfer to other work or quantify run-to-run variability.","watch_conditions":["Watch for replicated comparisons across task types and repeated runs that separate model, effort, orchestration, and team-size effects."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this creates a distinct analytical record around measured agent-team economics and benchmark boundaries."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/do-agent-teams-pay-off-a-case-study-04444a/","markdown":"https://newruntime.com/posts/do-agent-teams-pay-off-a-case-study-04444a.md","json":"https://newruntime.com/posts/do-agent-teams-pay-off-a-case-study-04444a.json"}
---

# Do Agent Teams Pay Off? A Case Study on Vibe Code Bench (18 minute read)

## Retrieval answer

**Thesis:** Agent teams should be judged by verified task outcomes relative to their added cost and time, not by parallelism alone.

**Thesis:** Agent teams should be judged by verified task outcomes relative to their added cost and time, not by parallelism alone. **Mechanism:** Delegation creates additional working contexts and orchestration overhead; repeated cached input for subagents drove much of the extra spend, while orchestration style affected whether parallelism reduced runtime.

Vals AI tested GPT 6 Sol and Claude Opus 5.5 on 50 Vibe Code Bench apps, comparing single agents and teams at medium and maximum reasoning effort. Each of the eight configurations ran once. Teams cost 1.8 to 5.1 times as much as single agents, and only GPT 6 Sol at medium effort showed a statistically significant team score gain: 7.3 points (84.9% versus 77.6%, p = 0.005). **Why now:** Both OpenAI and Anthropic now offer native multi-agent support, making the operational choice between delegation and a stronger single-agent run immediate for teams building software with these systems.

**First implication:** Benchmark delegation against higher reasoning effort on representative tasks, tracking score, cost, and runtime together. **Evidence boundary:** The study covers one benchmark of full-stack web apps, 50 tasks, and one run per configuration; results may not transfer to other work or quantify run-to-run variability. The leads also orchestrated differently: Sol generally split work early and tested, while Opus often worked through sequential waves, illustrating that team behavior is not a single treatment. **First watch condition:** Watch for replicated comparisons across task types and repeated runs that separate model, effort, orchestration, and team-size effects.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)