Rather than grading an agent’s prose or tool-call format, it checks terminal backend state and side effects after a task.
ThinkingBox contains 507 workflows across retail, auto insurance, travel, neobanking, and consulting. Each task is run 20 times from an identical clean backend. The release reports pass@1, whether a task succeeds at least once in 20 attempts, and the literal count of tasks that pass all 20 runs.
In a 121,680-trial common-set analysis across 12 models, 79,853 attempts failed executable checks. Of those failures, 67.24% ended cleanly, used a state-changing tool, and reported no final tool error. The authors also report model-specific consistency and estimated cost measures; for example, Kimi-K3 solved 476 of 507 tasks at least once but passed all 20 runs on 68 tasks.
The source says the benchmark can be run through OpenEnv. It does not establish how these results predict reliability in a particular production stack, nor does it provide a production-service price or guarantee of agent performance.