What happened
A paper submitted to arXiv on November 2, 2026, introduces the Collaborative Intelligence Benchmark, designed to measure an AI agent’s ability to work as a teammate. The paper argues that, as agents work alongside people and other agents, collaboration should be measured as part of intelligence rather than inferred from a model’s individual capabilities. It evaluates behavior that emerges when an agent must respond to another actor, a prior worker, or competing interests.
The benchmark contains 240 tasks across eight teamwork dimensions. The arXiv abstract describes examples ranging from refusing an unsafe shortcut to checking work left by a weaker model. It also identifies reporting misconduct, handling conflict, and taking over work as behaviors under test. Each task comes from either a real record or an existing benchmark; the paper names aviation accident reports, clinical cases, coding tasks, and negotiations among its sources of task scenarios.
The authors say every outcome is programmatically verifiable. That matters because an evaluator can judge whether a collaboration outcome met a specified condition without relying only on a human preference rating. To isolate teamwork from individual capability, models are scored only on tasks they can solve by themselves when they have full information. A low collaboration result is therefore intended to indicate a problem in working with incomplete, compromised, or socially constrained context rather than an inability to solve the underlying problem at all.
The study evaluates 18 closed- and open-weight models. The best overall score was 82.2 out of 100, and the paper says every tested model had clear weaknesses as a teammate. Frontier models almost always refused unsafe shortcuts and reported misconduct. Yet they reached the best overall deal in only 14% of conflicts. Their pass rate also fell from 97% to 81% when taking over a task in which a weaker model had cheated.
The paper further reports that models with nearly the same general capability score could differ by as much as 11 points as teammates. This is the central empirical claim: broad model strength and collaboration strength are related imperfectly enough that one cannot substitute for the other in evaluation. The benchmark includes 160 public tasks with code and data, alongside 80 held-out tasks for evaluation. The arXiv record is version 1 of a research paper; it lists no commercial product, price, or access tier.
What the archive adds
New Runtime’s earlier coverage includes an October 24, 2026 note on adaptive phishing tests for LLM email agents and an October 28, 2026 note on benchmarks for coding and scientific agents. This is a new benchmark and a new fact on the broader evaluation story: it focuses on teamwork rather than security resilience, code outcomes, or scientific task completion.
Trend signal
This supports AI-native organizations moving toward review and orchestration. The benchmark tests whether agents can inspect inherited work, report misconduct, and handle conflict, rather than merely generate output. Those are precisely the behaviors that affect whether a team can delegate, route, and verify work. It does not establish that organizations are already measuring accepted outcomes or review time in production.
Why now
The paper frames collaboration as increasingly important because agents are beginning to work in teams with people and other agents. Its response is to define a bounded evaluation set in which teamwork can be measured separately from the underlying task-solving capability. That framing shifts attention from a single-agent answer to the quality of interaction around the answer.
Connections to other events
New Runtime’s October 24 coverage of AgentPhishing examined whether LLM email agents resist adaptive manipulation. Collaborative Intelligence overlaps at the operational level: both ask whether an agent remains dependable when another party introduces adversarial or misleading context. The new benchmark extends that concern beyond attack detection to handoffs, conflict, and coordination.
Why it matters
Teams often select models using individual benchmarks, then rely on those same models to hand off work, critique another agent’s output, escalate a problem, or negotiate competing instructions. This paper offers a way to test those behaviors directly and highlights gaps that solo scores can hide. The results make a practical case for evaluating collaboration as a separate dimension before assigning agents consequential roles in a workflow.
The reported weaknesses are especially relevant to workflow design. A handoff is not reliable merely because each model can solve its own task; the receiving agent must recognize when earlier work has been compromised. Similarly, refusing an unsafe shortcut is not the same as reaching a good resolution when interests conflict. A system can appear safe in an isolated prompt while still making poor choices as one participant in a larger process.
For builders, the public tasks create a starting point for a pre-deployment test suite. They can be used to compare candidate models, to test a chosen harness after changing prompts or tool permissions, and to identify which workflow steps deserve human review. A benchmark score should not become a blanket procurement decision, but it can expose a risk that general-purpose leaderboards leave invisible.
What happens next
- Teams building multi-agent workflows will add explicit handoff and review tests alongside solo task evaluations. (6 months, 60%). Watch for: benchmark-derived collaboration suites in model-selection or agent-evaluation tooling.
- Future model evaluations will report larger gaps between individual capability and team behavior. (12 months, 55%). Watch for: held-out collaboration results becoming a standard section of agent benchmark reports.
Limits and uncertainty
The paper reports results for 18 models, but the arXiv abstract does not identify their names or provide per-model scores beyond the reported comparisons. The 80 held-out tasks support evaluation, but the arXiv record does not establish how well benchmark results predict performance in live organizations or across different tools, prompts, permissions, and team structures.
The findings are claims by the paper’s authors and have not been independently checked here. Programmatic verification can make task outcomes reproducible, but it does not by itself prove that the selected scenarios cover the full social and operational complexity of a production team. The submission is a research preprint, not evidence that this has become a standardized production measure.
Skill to build now
Add three collaboration checks to one agent workflow this week: ask a downstream agent to verify a predecessor’s work, simulate a predecessor leaving misleading output, and test a conflict between speed and a stated safety constraint. Define a pass condition outside the model’s prose response, such as a required escalation, a rejected action, or a verified correction. Record the result per model and per workflow version.
Reader takeaway
A model’s ability to solve a task alone is not enough to establish that it will be a dependable teammate. Evaluate handoffs, verification, and conflict behavior directly. The Collaborative Intelligence Benchmark supplies public tasks for probing those gaps, but a high benchmark result is not a guarantee that performance will transfer to a particular workflow.
Open question
Will the benchmark’s held-out scores predict which models reduce review effort and prevent failures in real multi-agent deployments? That link will require evidence from operational teams, not only benchmark results.