{"type":"post","id":"arena-ai-replaced-bash-recovery-with-bash-reliability-0fc075","slug":"arena-ai-replaced-bash-recovery-with-bash-reliability-0fc075","title":"Arena.ai replaced Bash Recovery with Bash Reliability , scoring first-try shell command accuracy.","description":"**Thesis:** Agent evaluation becomes more interpretable when a metric states which errors it attributes to the agent and which it excludes.","retrieval_nugget":"**Thesis:** Agent evaluation becomes more interpretable when a metric states which errors it attributes to the agent and which it excludes.","published_at":"2026-10-14","updated_at":"2026-10-14","record_date":"2026-10-14","date_kind":"published_at","topics":[],"source_urls":["https://arena.ai/resources/agent-arena-signals"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:arena-ai-replaced-bash-recovery-with-bash-reliability-0fc075","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:5f324d0b791d172517f9cce3a8262bccc920897837e0440254e7b62803f1567f","reviewed_at":"2026-10-14T05:02:39.026Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Agent evaluation becomes more interpretable when a metric states which errors it attributes to the agent and which it excludes.","observed_facts":[{"text":"Arena.ai states that Bash Recovery is now called Bash Reliability.","source_urls":["https://arena.ai/resources/agent-arena-signals"]},{"text":"The source classifies bash calls as success, failure, or exempt; agent-caused errors count as failures and external errors such as network outages are exempt.","source_urls":["https://arena.ai/resources/agent-arena-signals"]},{"text":"The score is based on failures per non-exempt call; correctly run tests that find issues and searches that return no results count as successes.","source_urls":["https://arena.ai/resources/agent-arena-signals"]}],"mechanism":"The signal attributes failures to the agent when it causes an error, excludes errors outside its control, and computes failures per non-exempt bash call.","why_now":"Arena.ai identifies these definitions as the ground truth for turning live interactions into model-level Agent Arena signals, making the scoring rule relevant to interpreting its leaderboard.","implications":["Readers can distinguish model-attributed command failures from environmental failures when interpreting the signal."],"evidence_boundary":"The supplied public page defines the metric but provides no model scores, comparative results, or evidence that the change alters rankings.","watch_conditions":["Watch for Arena.ai to publish model-level Bash Reliability results or explain how the signal affects leaderboard comparisons."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this distinct record captures the documented Bash signal rename and its scoring rules."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/arena-ai-replaced-bash-recovery-with-bash-reliability-0fc075/","markdown":"https://newruntime.com/posts/arena-ai-replaced-bash-recovery-with-bash-reliability-0fc075.md","json":"https://newruntime.com/posts/arena-ai-replaced-bash-recovery-with-bash-reliability-0fc075.json"},"next_reads":[{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}