Watch condition: Watch for replications that test whether the reported model-by-harness interactions persist on real engineering workloads and newer coding models.
Thesis
For production coding agents, harness design should be treated as a model- and task-dependent control surface, not a fixed wrapper around a model.
Observed facts
The authors report 176 matched settings across four models, evaluated on SWE-Bench Verified and Terminal-Bench 2.1.
1They find context management more valuable as the context-window budget tightens, with most benefit coming from preventing context-overflow failures.
1They also report that planning becomes a cost saver for stronger models with little accuracy change, while predefined tools help models with weaker bash proficiency.
1
Mechanism
Context management prevents overflow failures; planning changes where execution trajectories stop; and action-space design changes the granularity at which code is written.
Why now
As coding agents are compared across models and tasks, aggregate scores can obscure whether gains come from model capability or the harness choices surrounding it.
Implications
Teams should test harness components against their actual model and task mix rather than assume a single tool interface is universally best.
Cost and accuracy should be read together, especially when selecting a shell-only interface or adding planning and context-management steps.
Evidence boundary
These are findings reported in the abstract of an arXiv preprint; they do not establish independent replication or general production performance beyond the tested configurations and benchmarks.
Watch conditions
Watch for replications that test whether the reported model-by-harness interactions persist on real engineering workloads and newer coding models.