What happened
IBM and Kaggle announced a leaderboard for ScarfBench, the Self-Contained Application Refactoring Benchmark. ScarfBench evaluates whether a coding-agent system can migrate enterprise Java applications between Jakarta EE, Quarkus, and Spring while leaving applications able to build, start, deploy, and retain expected behavior. IBM introduced the benchmark earlier in 2026; the new collaboration makes it a continuously maintained comparison rather than only a published evaluation.
Kaggle consulted on the evaluation design, independently reproduced and ran ScarfBench on its platform, sponsored the required compute, and will maintain the benchmark as models are released. The reported systems are model-and-harness combinations, rather than measurements of a language model alone. The harness determines how the model works with a repository and development environment, so a rank should not be read as an intrinsic model ranking.
Across 204 migration tasks, Gemini 4 Argon with Antigravity ranked first on the primary smoke-test score at 75.2%. Claude Opus 5.5 with Claude Code followed at 70.5%, Claude Opus 5 with Claude Code at 69.8%, and GPT-6.1 Sol with Codex at 68.4%. Ten configurations were evaluated. The primary metric gives partial credit and weights larger test suites more heavily. On a separate all-tests-pass measure, Claude Opus 5.5 with Claude Code led at 72.1%, narrowly above Gemini 4 Argon with Antigravity at 71.1%.
A sensitivity analysis excluded four tasks for which every configuration earned zero behavioral-test credit. Scores rose, but the ordering at the top did not: Gemini 4 Argon with Antigravity reached 79.4%, Claude Opus 5.5 with Claude Code 74.5%, and Claude Opus 5 with Claude Code 73.7%. Even the best result left considerable room for improvement: no configuration completely passed much more than roughly three quarters of migrations.
The source’s most consequential finding is the separation between compilation and behavioral fidelity. After excluding the four universally unsolved tasks, eight of ten configurations compiled at least 97% of tasks, while smoke-test performance varied by more than 50 points. Of 745 classified failing runs, 535, or about 72%, were runtime behavioral failures. These included assertion or content mismatches, lost routes and 404 responses, and explicit HTTP 500 or 405 errors. Compile or packaging failures accounted for roughly 20%, and boot or dependency-injection crashes about 8%. ScarfBench’s public submission flow uses a pull-request-based pipeline.
What the archive adds
This is a new leaderboard event on a known editorial thread: Artificial Analysis’ FrontierCode 1.1 leaderboard similarly treated coding-agent comparison as task- and cost-aware, while noting that every leaderboard remains bounded by its setup. ExtractBench Pro’s Gemini 4 Argon results provided earlier Argon-specific evaluation data, but for extraction rather than enterprise application migration.
Trend signal
This supports Verification bandwidth is the scarce engineering resource. The benchmark shows why output volume and compilation are inadequate proxies for useful agent work: the hard verification problem is whether a migrated service still behaves correctly after deployment. Its failure breakdown makes runtime validation, tests, and review capacity central constraints on coding-agent adoption.
Why now
IBM says ScarfBench was introduced earlier in 2026 as a reproducible evaluation. The Kaggle partnership adds independent reruns, sponsored compute, and planned maintenance as new models arrive, changing the benchmark from a fixed publication into an updating comparison surface.
Connections to other events
The FrontierCode leaderboard covered a different coding-agent benchmark, but its archive note also framed comparison around task outcomes and cost rather than a model name alone. ScarfBench makes the same unit of analysis explicit: the evaluated object is the combined model and coding harness, then tested against a deployed application’s behavior.
Why it matters
Teams considering agents for legacy modernization should treat a clean build as an intermediate artifact, not a completion condition. Framework migration changes dependency injection, persistence, endpoints, build configuration, descriptors, and deployment infrastructure together. A system can produce plausible code and still lose a route, retain source-framework wiring, or package an application incompatibly with its target runtime.
The leaderboard is useful because it tests that full path and identifies a reproducible comparison method. It is also a reminder to evaluate the exact operating configuration a team intends to use. Changing the harness changes repository interaction and therefore the observed outcome. Procurement or pilot criteria should include build, boot, deployment, behavioral suites, and the human capacity to investigate the failures that remain.
What happens next
- Maintained submissions will add newer model-and-harness combinations to the comparison. (6 months, 75%). Watch for: new independently reproduced leaderboard runs.
- Teams piloting migration agents will place more weight on runtime and behavioral acceptance tests than compilation rates. (12 months, 70%). Watch for: evaluation reports that separate build success from deployed-test success.
Limits and uncertainty
ScarfBench is a benchmark of 204 specific migration tasks across Jakarta EE, Quarkus, and Spring, not a measurement of all software modernization work. The reported scores apply to named model-and-harness pairings and cannot establish how either component would perform in a different harness, repository, security environment, or organization. The source describes independent reproduction and a maintained leaderboard, but it does not establish production reliability, total project cost, or the amount of human review required for a real migration. Smoke scores also award partial credit, so they should be read alongside the all-tests-pass result and failure categories.
Skill to build now
Create a migration acceptance harness for one representative service this week. Require the agent’s branch to build, boot in the target environment, expose expected routes, and pass behavior-focused tests before it is considered successful. Record failures by category—compile, startup, deployment, route, and assertion mismatch—so that a pilot reveals whether its bottleneck is code generation or behavioral verification.
Reader takeaway
The meaningful question for a migration agent is not whether it can rewrite Java code or make a project compile. It is whether the migrated application still works after deployment. ScarfBench provides a public, maintained way to compare that outcome across complete model-and-harness systems, while showing that runtime behavior remains the main gap.