The accompanying implementation is available on GitHub, according to the joint Hugging Face post.
ThinkingBox-Bench contains 507 tasks. Each task starts a backend, gives an agent MCP tools and a simulated user with private context, then checks database records after the run. The reward is based on whether the required terminal state is correct, rather than whether the agent followed a prescribed tool-call sequence.
The authors selected training tasks outside the benchmark using a one-time estimate of difficulty for the base policy, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. On the 507-task held-out benchmark, pass@1 rose from 51.70% to 60.78%. Tasks that passed all 20 recorded attempts rose from 38 to 107, while tasks solved at least once fell from 453 to 446.
The source reports a trade-off toward repeatable success rather than broader at-least-once coverage. It does not establish independent replication, general performance beyond this benchmark, or total deployment economics: its cost comparison excludes training and policy-probe costs.