What happened
In a November 3, 2026 Hugging Face article, IBM Research describes a controlled experiment comparing three agent harnesses: IBM’s CUGA (ConfigUrable Generalist Agent), Nous Research’s Hermes, and LangChain’s Deep Agents. The authors used GLM-5.3-flash for all three, running each on the same 90 AppWorld tasks across 30 scenarios, with one attempt per task. AppWorld covers multi-step work across nine apps and 457 APIs, including email, payments, and file storage. The authors used the Harbor evaluation framework and published their setup in an IBM Harbor fork.
CUGA passed 73 of 90 tasks, or 81.1%; Hermes passed 42, or 46.7%; and Deep Agents passed 27, or 30.0%. The article also reports scenario-level consistency: CUGA passed all three variants in 20 of 30 scenarios (66.7%), Hermes did so in eight (26.7%), and Deep Agents in two (6.7%). IBM says this scenario measure is more informative than a task total because a scenario counts as passed only when all three variants pass.
The compared configurations were not identical. CUGA used AppWorld instructions, task decomposition, verification before execution, and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP, LangChain middleware configured for 469 tools, including tool selection, context summarization, a to-do list, retries, and call limits; IBM says it also applied two small fixes to prevent malformed GLM replies from ending runs. The experiment is therefore a comparison of these particular configurations, not an isolated measurement of each harness’s inherent capability.
IBM presents CUGA as an open-source harness for enterprise work across business systems. The described loop prepares, calls the model, executes, and repeats; its configuration can include task decomposition, focused tools, Python-based API calls in a restricted sandbox, checks before state-changing code executes, and reflection afterward. The article says CUGA is Apache 2.0, works with OpenAI-compatible endpoints, and can run locally, on Kubernetes in private or sovereign clouds, or in air-gapped clusters. It also describes policy mechanisms including human approval gates and tool guards that can block calls. The article gives no price for CUGA.
The authors explicitly caution that AppWorld is only a proxy for some enterprise automation capabilities, the sample contains 90 tasks on one benchmark, configurations differed, and the runs did not use policies—so the benchmark does not test policy enforcement. IBM recommends other tools for other jobs: terminal coding agents such as Codex or Claude Code, personal always-on assistants such as Hermes, and LangGraph for a blank-canvas graph framework.
What the archive adds
A prior Field Note, Cline Agents Harness Unlocks Benchmark-Leading Performance from NVIDIA Nemotron 3.5 Super VL, reported a harness-specific Terminal-Bench result. That was a different model and benchmark; this is a new comparison and adds evidence about harness variation, not a directly comparable score. Earlier Telegram coverage also discussed Codex and Pi harness behavior and Hermes profiles, but this IBM report brings a controlled same-model comparison and enterprise-oriented selection framing.
Trend signal
This supports the harness architecture outlives model choice pattern: holding GLM-5.3-flash constant while changing the harness coincided with large performance differences on AppWorld. It is evidence for the importance of harness fit in this workload, not proof that models are generally interchangeable or that these results transfer to other tasks.
Why now
IBM’s article situates the comparison amid a crowded field of nearly 200 agent harnesses and argues that builders should select for workload fit rather than look for a universal winner. The benchmark offers one concrete enterprise-automation case against that backdrop, while also inviting competing configurations to be rerun on the same tasks.
Why it matters
Teams evaluating agents often focus on model choice, but the comparison shows that planning, tool exposure, verification, retries, and reflection can coincide with different completion outcomes even when the model is fixed. For multi-app business automation, those harness choices also affect how a system handles state changes and whether it can be configured around organizational rules.
The practical implication is to treat benchmark performance as a property of a model-harness-task configuration. A result on AppWorld can inform a shortlist, but it does not establish that CUGA will outperform Hermes or Deep Agents on a company’s workflows. Teams should reproduce relevant tasks with their own tools, permissions, and failure conditions, and evaluate consistency as well as aggregate task completion. Policy claims need separate testing: IBM’s benchmark did not run policies, even though policy enforcement is a central part of its product argument.
Limits and uncertainty
The evidence comes from IBM Research, which develops CUGA, and is not an independent evaluation. It covers 90 tasks on one benchmark, with one attempt per task. AppWorld is described as a proxy, not a complete enterprise environment. The setups differ in tools and configuration, including fixes applied to Deep Agents. The experiment does not show whether the results generalize to other models, task families, or production workloads; nor does it test CUGA’s policy enforcement. No price or comparative deployment cost is stated.
Reader takeaway
The useful conclusion is not that one harness is universally best. On this AppWorld setup, changing the harness while keeping the model fixed produced a wide spread in task completion. Treat the harness as part of the system under test, choose one that fits the work, and reproduce results on your own workflows before making a production decision.