What happened
Fireworks’ AutomationBench page presents Zapier’s benchmark for business-operation agents. The evaluation covers 657 held-out workflows across sales, marketing, operations, support, finance, and HR. Each workflow takes place in a small simulated company spanning 47 SaaS applications. An agent receives one request, searches for the API endpoints it needs, follows instructions that may be stored in an inbox or spreadsheet, and makes changes across systems.
This setup tests a workflow rather than a single chat response. An agent may need to identify an appropriate service endpoint, interpret a rule located outside the initial request, and write the required changes into more than one application. The task structure therefore combines retrieval, tool selection, rule-following, and execution. Fireworks says the benchmark uses held-out tasks, meaning the 657 workflows are the evaluation set represented in the table rather than examples selected from a model’s own output.
Scoring is based on the final state, not whether an agent produced a plausible explanation or completed most of a sequence. Deterministic checks test the result, and a task passes only if every assertion holds—including checks intended to catch errors such as sending a notice to the wrong team. A partially correct workflow is consequently not counted as a pass. The page describes the metric as a strict pass rate and says the displayed tasks represent one run.
The leaderboard was last updated November 13, 2026, and the page says it is checked daily. Gemini 4 Carbon leads at 53.9%, followed by Claude Fable 5.2 at 51.4% and Gemini 4 Argon at 51.3%. Gemini 4 Hydrogen is close behind at 51.1%. The top four are separated by 2.8 percentage points from first to fourth, while Carbon’s lead over Fable 5.2 is 2.5 points.
The score is paired with a listed cost per task: $2.12 for Carbon, $4.64 for Fable 5.2, $1.70 for Argon, and $0.82 for Hydrogen. The table also includes Claude Sonnet 5.5 at 44.7% and $1.14, Claude Opus 5.5 at 42.5% and $1.44, GPT-6 Astra at 41.4% and $1.73, and GPT-6 Sol at 33.2% and $0.27. The highest-ranked open model shown is Kimi K3, at rank 14 with a 22.7% strict pass rate and listed cost of $0.43 per task. Fireworks links readers to Zapier’s benchmark page and a methodology section.
The gap between leading results and universal reliability remains substantial: even the top score means that nearly half of the tested workflows did not satisfy every required check in this run. That is a meaningful distinction for automations that can update customer records, route requests, send notifications, or trigger work across several business systems.
What the archive adds
New Runtime’s earlier coverage, IBM and Kaggle launch Java framework migration leaderboard (October 31, 2026), covered a different coding-agent benchmark, ScarfBench, where Gemini 4 Argon led at 75.2%. This is a new benchmark result on business workflows, not a restatement: the tasks, scoring context, and reported cost measure differ.
Trend signal
This update supports agent economics moves to completed work: the leaderboard puts cost per task beside strict task completion, rather than treating model-call price or a general model score as sufficient. The result also shows why the unit needs careful qualification. Passing requires every final-state assertion to hold, and the leader clears just over half of the benchmark tasks.
Why it matters
For teams evaluating agents that act across business software, the choice is not simply which model scores highest. AutomationBench makes a useful distinction between completion quality and cost. Gemini 4 Carbon leads on strict pass rate, but Gemini 4 Hydrogen is only 2.8 percentage points behind and has a much lower listed task cost. A team whose workflow has a review queue may value that difference differently from a team where a missed condition creates a costly downstream error.
The benchmark’s all-checks-must-pass rule is also a practical design prompt. An agent that sends the right message to the wrong team, or updates most but not all connected applications, has not completed the automation. Builders can borrow this framing for internal evaluations: define required final-state assertions, include cross-system and routing errors, run representative end-to-end workflows, and record cost alongside completion. The table can help create a shortlist, but it cannot substitute for a test against a company’s own systems, policies, and failure costs.
Limits and uncertainty
The leaderboard reports 657 held-out tasks and one run, so the scores do not show run-to-run variability or confidence intervals. A narrow difference between models may not translate into a stable difference under repeated execution. The page’s strict metric is useful, but the reported results do not establish how performance varies by business function, application, task difficulty, or individual failure mode.
The listed per-task prices are benchmark figures. Production cost can differ with prompt design, retries, tool calls, error handling, model configuration, and deployment conditions. Nor do these results independently establish performance on live company data, a team’s particular SaaS permissions, or the operational impact of a failed automation. A strict pass rate measures the benchmark’s defined end state, not every risk an organization may need to manage.
Skill to build now
Choose one internal workflow that writes to more than one system and turn its success definition into explicit end-state assertions. Include a check for the intended record changes, a check for the correct recipient or routing destination, and a check that no required system was skipped. Run candidate models through the same workflow more than once, then compare strict completion and cost per completed task rather than comparing response quality alone.
Reader takeaway
Read the leaderboard as a structured comparison, not a deployment verdict. Its most useful signal is the pairing of strict end-state success with cost per task: that exposes trade-offs hidden by a single rank. Before choosing a model for automation, replay representative workflows and define checks for every consequential state change.