Each bash call is classified as a success, failure, or exempt. Agent-caused errors, such as a mistyped flag or a bug in an agent-written script, count as failures; external errors such as network outages are exempt. Tests that run correctly but reveal problems, and searches that return no results, count as successes. The model score is based on failures per non-exempt call, with more failures lowering its Net Improvement score. The supplied source defines the metric but does not report model scores, rankings, or the effect of the change on leaderboard outcomes.
Raw SignalOct 14, 20261 min read1 source
Arena.ai replaced Bash Recovery with Bash Reliability , scoring first-try shell command accuracy.
In one minute
Arena.ai has renamed its Bash Recovery signal to Bash Reliability and published a definition of how it is scored.