---
type: analysis
slug: research-the-best-task-solver-is-not-the-best-coach
title: "The best task-solver is not the best coach"
description: "CentaurBench, a July 2026 UC Berkeley Haas working paper, finds only a modest link between a model’s ability to finish professional work and its ability to coach a weaker worker. Independent human studies in tutoring, coding, medicine, and occupational collaboration show the same split, but CentaurBench itself does not measure humans."
published_at: 2026-10-25T02:00:00.000Z
topics: ["automation","claude","agents","research"]
source_urls: ["https://arxiv.org/html/2608.18554v1","https://www.nber.org/papers/w35663","https://kennywong524.github.io/centaur-benchmark/","https://raw.githubusercontent.com/kennywong524/best-player-not-best-coach/main/README.md","https://arxiv.org/html/2506.05579v2","https://arxiv.org/html/2504.18919","https://www.nature.com/articles/s41562-024-02024-1","https://arxiv.org/abs/2606.09833","https://arxiv.org/html/2502.18940v2","https://arxiv.org/abs/2502.18940","https://github.com/kennywong524/best-player-not-best-coach"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
In late August 2026, a UC Berkeley Haas working paper argued that the skill of finishing professional work and the skill of helping someone else finish it are not the same thing. CentaurBench, posted as NBER Working Paper 35663 and arXiv:2608.18554, is the primary source behind that claim. Its authors, Pattaraphon Kenny Wongchamcharoen, Kris Gulati, Min Min Fong, and Abhishek Nagaraj, built a two-regime benchmark: models either complete seven economically grounded tasks themselves or write process guidance for a weaker worker model. The paper’s slogan, “the best player is not always the best coach,” is aimed at a real product choice. Labs and buyers now rank models on agentic, end-to-end work, then ship those same models as copilots.

That choice matters because most workplace AI is still used as help, not as a replacement. If automation leaderboards are a weak proxy for assistance quality, organizations that pick a single “best model” can install a strong soloist that is a poor coach—or, worse, a coach whose guidance makes the worker’s output worse. The research question is whether that split is real, whether it applies to humans rather than simulated workers, and whether the industry’s agent push makes cowork harder.

## What we checked

- **Human augmentation and task automation are not necessarily related.** partly confirmed. CentaurBench reports a Spearman rank correlation of 0.48 between automation and augmentation rankings across nine assistant models, with p=0.187, so the association is not statistically distinguishable from zero at conventional levels. The relationship is task-dependent, ranging from 0.85 on tax preparation to −0.04 on travel planning. [1][2][3]
- **Models that are really good at doing work are not always good at helping humans do work better.** partly confirmed. On five of seven CentaurBench tasks the automation winner is not the augmentation winner, including large reversals such as Claude-Opus-4.8 on market trends (mean rank 2.05 automating versus 8.15 assisting). That paper coaches a GPT-3.5-Turbo worker, not a human. Separate human studies find the same directional split: KITE’s N=118 coding and math study, Bean et al.’s N=1,298 medical-advice trial, MathTutorBench, and CollabSkill all show that strong solo or autonomous scores do not reliably predict help to people. [1][5][6][9][8]
- **Pressure to make models good agents may undermine human-AI cowork.** unresolved. No opened source shows that agent or RL-for-agents training causally degrades coaching. CentaurBench and CollabSkill argue that labs optimizing autonomous leaderboards will miss assistance quality, and MathTutorBench reports a pedagogy-versus-solving trade-off, but those are ranking and specialization results, not evidence that agent pressure undermines cowork. [1][8][9][5]
- **A primary paper exists that separates automation performance from measured human augmentation.** partly confirmed. The primary work is CentaurBench (arXiv:2608.18554, NBER Working Paper 35663), which jointly ranks models as automators and as process coaches of a fixed weaker model. It does not measure humans. Human augmentation is instead measured in separate studies (KITE, Bean et al., Vaccaro et al., CollabSkill). [1][2][11]
- **CentaurBench’s ‘augmentation’ condition is one-shot process coaching of GPT-3.5-Turbo, not interactive human cowork.** confirmed. The paper and dashboard state that assistants write process-only assistance text for a fixed GPT-3.5-Turbo worker and are forbidden from producing the deliverable. The authors call this a conservative lower bound on augmentation and list human validation as future work. [1][3][4]
- **Assistance is not reliably positive and can leave the worker worse off than working alone.** confirmed. The unaided GPT-3.5-Turbo worker outranks every assisted condition on operations research, tax preparation, and travel planning. Averaged across seven tasks, only GPT-5-Mini (mean rank 3.66) beats the unaided baseline (3.79). [1][2][3]

## What we found

- CentaurBench evaluates ten models on seven professional tasks drawn mainly from GDPval and the Anthropic Economic Index: counseling, market-trends analysis, meal planning, operations research, tax preparation, travel planning, and tutoring. In automation mode the focal model writes the deliverable. In augmentation mode it writes a three-phase process plan for GPT-3.5-Turbo, which produces the deliverable. Blind pairwise LLM judges with task-specific rubrics and leave-one-family-out masking score ten independent runs. Rankings across regimes correlate only modestly (Spearman ρ=0.48, p=0.187). The automation winner loses the augmentation contest on five of seven tasks. [1][2][3]
- Role reversals are large and stable across runs. Claude-Opus-4.8 is the strongest automator on market trends (mean rank 2.05) and among the weakest assistants (8.15). GPT-4.1 is a weaker direct solver on counseling (7.40) and the strongest assistant (3.80). GPT-5-Mini is the only model that ranks first on average in both regimes, and it is the only assisted condition whose average augmentation rank (3.66) beats the unaided worker (3.79). Cross-mode correlation is high on tax preparation (ρ=0.85, p=0.004) and essentially zero on travel planning (ρ=−0.04). [1][3]
- The authors inspect assistance texts and argue that bad coaching is specific: restating the prompt, failing to impose structure, or instructing the worker not to produce the required analysis. They treat the unaided-worker wins on three tasks as evidence that poorly matched or overly complex process guidance can be worse than none. They also warn that their automation ranking is for single-turn, no-tool deliverables and should not be read as an agentic-benchmark ranking. [1]
- CentaurBench is a simulation, not a human field experiment. Judges agree on 71.0 percent of comparisons (Krippendorff’s α=0.414), with weaker agreement in augmentation (67.8 percent, α=0.354) than automation. The authors present the study as a pilot and call for human-expert validation, multi-turn workflows, and varied worker models. The public repo (kennywong524/best-player-not-best-coach) and dashboard release the ten-run outputs used in the paper. [1][4][3]
- Independent human studies support the directional claim. In KITE (N=118), Gemini-2.5-Pro’s 81.3 percent solo code accuracy coincided with a 10.0 percentage-point drop in collaborative efficacy, while Claude-3.7-Sonnet and GPT-4o rose 25.0 points from 45.0 percent solo. Bean et al. (N=1,298) found LLMs alone identified relevant conditions in 94.9 percent of medical scenarios and the correct disposition in 56.3 percent, but people using those models identified conditions in under 34.5 percent of cases and dispositions in under 44.2 percent, no better than a control that mostly used search. Vaccaro, Almaatouq, and Malone’s meta-analysis of 106 experiments found human–AI teams beat humans alone (Hedges’ g=0.64) but lost to the better of human or AI alone (g=−0.23), with losses concentrated when AI already outperformed the human. [5][6][7]
- The split also appears in agent products and tutors. CollabSkill paired 93 workers with agents across 386 occupational sessions and ranked Claude Code above Codex (CollabSkill 74.778 versus 71.267), reversing Codex’s lead on the authors’ solo run and on SWE-bench Verified, GDPval, and SciCode. Claude Cowork, sharing Claude Code’s model, ranked higher still (76.738), isolating interface. MathTutorBench finds that math solving does not translate to scaffolding: Qwen2.5-Math-7B-Instruct scores 0.88 on problem solving and 0.06 on scaffolding win rate. [8][9][10]

## What it means

Builders should stop treating a single automation or agent leaderboard as the model-selection rule for copilots, tutors, and multi-agent systems. If the product is a coach, the evaluation has to score the downstream worker’s deliverable under that coach, not the model’s solo output. CentaurBench, CollabSkill, and KITE all show rank reversals once the model is placed in an assistive role, and CentaurBench shows that process-only guidance can be worse than silence. The practical response is role-specific eval: keep an automation suite for agents that own the deliverable, add an augmentation suite that holds the worker fixed, and, for human-facing products, run the human study rather than a simulator.

Do not read this literature as proof that agent training is harmful to cowork. It is proof that cowork is a different capability and is currently under-optimized. If you train only on verified end-to-end rewards, you should expect coaching quality to be an accident. If you ship one model to every team, you should expect some teams to be coached into worse work. Route by job: solvers for closed, repeatable production; coaches selected on measured lift for the actual worker, interface, and task.

## Still open

- Does CentaurBench’s automation–augmentation split replicate when the worker is a human professional rather than GPT-3.5-Turbo, and when judges are domain experts rather than an LLM panel?
- Would richer assistance—drafting, iterative critique, or tool use—turn the currently negative coaching conditions positive, or would rank reversals remain?
- Is there a causal effect of agent-style or RL-with-verified-rewards post-training on coaching quality, as opposed to a mere ranking divergence across separately trained models?
- How should organizations combine models by subtask, given that GPT-5-Mini led both CentaurBench averages while other models swapped roles by domain?

## Sources

1. [CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks (arXiv HTML)](https://arxiv.org/html/2608.18554v1) (primary)
2. [CentaurBench NBER Working Paper 35663](https://www.nber.org/papers/w35663) (primary)
3. [CentaurBench interactive results dashboard](https://kennywong524.github.io/centaur-benchmark/) (primary)
4. [CentaurBench repository README](https://raw.githubusercontent.com/kennywong524/best-player-not-best-coach/main/README.md) (primary)
5. [When Models Know More Than They Can Explain (KITE)](https://arxiv.org/html/2506.05579v2)
6. [Clinical knowledge in LLMs does not translate to human interactions](https://arxiv.org/html/2504.18919)
7. [When combinations of humans and AI are useful: a systematic review and meta-analysis](https://www.nature.com/articles/s41562-024-02024-1)
8. [CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks](https://arxiv.org/abs/2606.09833)
9. [MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors](https://arxiv.org/html/2502.18940v2)
10. [MathTutorBench arXiv abstract](https://arxiv.org/abs/2502.18940)
11. [CentaurBench reproducibility repository](https://github.com/kennywong524/best-player-not-best-coach) (primary)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Automation](https://newruntime.com/topics/automation.md): Explore the automation topic hub.
- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)