ScarfBench evaluates model-and-agent-harness combinations for enterprise Java migration, pairing rigorous migration validation with independently reproduced runs and a Kaggle leaderboard.
ScarfBench (Self-Contained Application Refactoring Benchmark) is an open IBM benchmark for evaluating whether coding agents can successfully migrate enterprise Java applications across frameworks while preserving working application behavior. Across Jakarta EE, Quarkus, and Spring migrations, it tests whether generated applications build, start, and continue to behave as expected, using both focused workloads and whole applications
Modernizing an enterprise application is not a single-file coding task. Moving an application between frameworks can require coordinated changes to dependency injection, persistence, database access, web endpoints, build systems, framework descriptors, and deployment infrastructure. A coding agent can produce a migration that looks plausible, or even compiles, while the application still fails when deployed or exercised against its expected behavior.
ScarfBench asks a practical question: Can an agent take an unfamiliar application, migrate it across framework boundaries, and leave behind software that still works?
IBM introduced ScarfBench earlier in 2026 as a reproducible benchmark for AI-driven migrations. Now IBM and Kaggle are partnering to make that evaluation easier to reproduce, compare, and extend over time. Kaggle consulted on the evaluation design, independently reproduced and ran ScarfBench on its platform while sponsoring the compute, and will maintain the benchmark as new models are released. The collaboration turns ScarfBench from a published benchmark into a continuously updated view of progress in agentic software modernization.
The original ScarfBench announcement described the benchmark design, developer-verified workloads, and reproducible evaluation approach in detail. This post focuses on what the latest evaluation tells us about model-and-agent-harness combinations, where they fail, and how the benchmark is evolving with Kaggle.
Evaluating model-and-harness combinations #
ScarfBench evaluates an agentic system rather than a language model in isolation. Each result reflects a model paired with a coding-agent harness that determines how it interacts with the repository and development environment. The results should therefore be read as model/harness results, not as measurements of the underlying model alone.
Across 204 migration tasks, Gemini 4 Argon with Antigravity ranked first on the primary smoke-test metric at 75.2%, followed by Claude Opus 5.5 with Claude Code at 70.5% and Claude Opus 5 with Claude Code at 69.8%. Ten model-and-harness configurations were evaluated.
| Rank | Model | Harness | Smoke % | All pass % | Mean % |
|---|---|---|---|---|---|
| 1 | Gemini 4 Argon | Antigravity | 75.2 | 71.1 | 85.9 |
| 2 | Claude Opus 5.5 | Claude Code | 70.5 | 72.1 | 82.0 |
| 3 | Claude Opus 5 | Claude Code | 69.8 | 60.8 | 79.7 |
| 4 | GPT-6.1 Sol | Codex | 68.4 | 62.7 | 76.8 |
| 5 | Gemini 3.8 Flash | mini-swe | 66.0 | 62.3 | 77.6 |
| 6 | GPT-6 Astra | Codex | 65.4 | 61.4 | 75.4 |
| 7 | Qwen 3.8 Max | qwen-code | 63.7 | 59.3 | 76.1 |
| 8 | Kimi K3 | kimi-code | 61.1 | 50.7 | 71.2 |
| 9 | Gemini 3.1 Pro | OpenHands | 39.5 | 35.1 | 50.5 |
| 10 | Gemma 4 26B-A4B | mini-swe | 24.5 | 17.6 | 30.8 |
The three metrics reveal an important nuance. The smoke score, the leaderboard metric, awards partial credit and gives larger test suites more weight. On that measure, the Gemini 4 Argon + Antigravity combination had the strongest result. But Claude Opus 5.5 + Claude Code had the highest all-tests-pass rate at 72.1%, narrowly ahead of Gemini 4 Argon. In other words, the Argon/Antigravity combination earned the most partial credit across the benchmark, while the Opus 5.5/Claude Code combination converted slightly more individual migration tasks into completely green test suites.
The ranking remains stable under a sensitivity analysis. Four of the 204 migration tasks received zero behavioral-test credit from every evaluated model/harness combination. Excluding those four tasks raised the absolute scores but did not change the ordering at the top: Gemini 4 Argon + Antigravity scored 79.4%, followed by Claude Opus 5.5 + Claude Code at 74.5% and Claude Opus 5 + Claude Code at 73.7%. This result suggests that the relative ranking is not driven by this small set of uniformly difficult tasks.
Even the strongest combination leaves substantial headroom: The best overall smoke score was 75.2% across the full benchmark, and no configuration fully passed more than about three-quarters of the migrations.
The biggest differentiator is behavioral fidelity, not compilation #
The clearest result is the gap between code that builds and software that behaves correctly. After excluding the four all-zero tasks, Claude Opus 5 + Claude Code, Gemini 4 Argon + Antigravity and Gemini 3.8 Flash + mini-swe each compiled 100% of tasks, GPT-6 Astra + Codex and Kimi K3 + kimi-code compiled 99.0% and 98.5%, and Claude Opus 5.5 + Claude Code and GPT-6.1 Sol + Codex compiled 98.0% and 98.5%. Yet smoke-test scores remained much lower.
Across the 10 configurations, compile success among solvable tasks spanned about 25 percentage points, but most configurations cluster near the top, with 8 of 10 compiling 97% or more; smoke-test performance spanned more than 50 points. Among applications that successfully deployed, the leading configurations clustered at roughly 63 to 76% for fully passing tests; Kimi K3 + kimi-code fell to 53%, Gemini 3.1 Pro + OpenHands to 42%, and Gemma 4 26B-A4B + mini-swe to 26%. In this evaluation, preserving behavior was a much stronger discriminator than getting the migrated application to compile.
Where agent configurations fail #
Of 745 classified failing runs, 535 (approximately 72%) were runtime behavioral failures: The application built and often booted but failed behavioral validation. This group included 304 content or assertion mismatches and 210 lost-route/404 failures; 21 were explicit HTTP 500/405 errors. Compile or packaging failures accounted for roughly 20% (152 runs), and boot or dependency-injection crashes for roughly 8% (58 runs).
Lost routes and HTTP semantics: In concurrency-taskcreator-quarkus-to-jakarta, Claude Opus 5 + Claude Code scored 0/6 after /taskinfo returned “File not found” and a test expecting HTTP 405 failed. In address-book-jakarta-to-quarkus, GPT-6 Astra + Codex scored 2/10 after list and create pages returned HTTP 404. #
Source-framework wiring surviving into the target: In cargotracker-jakarta-to-spring, Gemini 3.8 Flash + mini-swe retained JSF/JoinFaces wiring in the Spring Boot target, which then failed during startup while creating a facesContext bean. #
Packaging incompatible with the target runtime: In cart-spring-to-jakarta, GPT-6 Astra + Codex reached a deployment failure because the packaging type left by the migration was not supported by the target Liberty runtime.
These failures illustrate why ScarfBench validates beyond compilation: Nearly three in four classified failures in this evaluation appeared at runtime.
Agent efficiency reveals different operating profiles #
On the 200 tasks remaining after the four-task sensitivity exclusion, Claude Opus 5 + Claude Code used the fewest median turns at 32. GPT-6.1 Sol + Codex used 36, Gemma 4 26B-A4B + mini-swe 40, Gemini 4 Argon + Antigravity 42, GPT-6 Astra + Codex 45, Kimi K3 + kimi-code 47, Qwen 3.8 Max + qwen-code and Gemini 3.1 Pro + OpenHands 53, Claude Opus 5.5 + Claude Code 64, and Gemini 3.8 Flash + mini-swe 79.
Where harness-reported cost was available — for the Claude Code, Codex, mini-swe and kimi-code harnesses; the Antigravity, qwen-code and OpenHands harnesses did not emit per-run cost — the configurations showed distinct cost profiles:
| Model / harness | Median turns | Turns, successful runs | Cost / successful run | Cost per solved task |
|---|---|---|---|---|
| Claude Opus 5 / Claude Code | 32 | 30 | $1.76 | $3.83 |
| Claude Opus 5.5 / Claude Code | 64 | 60 | $5.98 | $10.10 |
| GPT-6.1 Sol / Codex | 36 | 34 | $0.64 | $1.25 |
| GPT-6 Astra / Codex | 45 | 44 | $3.55 | $6.65 |
| Gemini 3.8 Flash / mini-swe | 79 | 74 | $0.73 | $1.59 |
| Kimi K3 / kimi-code | 47 | 44 | $1.27 | $3.21 |
IBM and Kaggle are building a living ScarfBench leaderboard #
The latest ScarfBench evaluation is also the start of a deeper collaboration between IBM and Kaggle. The two teams bring complementary strengths: IBM developed ScarfBench around enterprise Java modernization, while Kaggle brings independent evaluation infrastructure and long-term leaderboard maintenance. The collaboration is intended to make the benchmark reproducible outside its original development environment and useful as new model-and-agent configurations are released.
The collaboration has three concrete parts. First, Kaggle consulted with IBM on the rigorous design of the benchmark and evaluation. Second, Kaggle independently reproduced and ran the evaluation on its platform and sponsored the compute. Third, Kaggle will maintain the benchmark long-term and add new models as they are released. The ScarfBench leaderboard provides the public view of these results. Public model submission is not available at launch but is on the roadmap.
What we have learned so far #
ScarfBench was built around the demanding premise that coding agents participating meaningfully in enterprise modernization need to be evaluated on the outcome enterprises ultimately care about: working software.
The latest evaluation reinforces that premise. Leading configurations can compile nearly every task in the sensitivity set, but compilation hides a harder problem. Routes disappear, HTTP behavior changes, source-framework wiring survives migrations, applications fail during startup, and business behavior diverges from the expected system.
That gap between code that builds and software that works is where ScarfBench is most informative. As new model/harness combinations arrive, a maintained benchmark can show whether progress comes from better generated changes, more effective debugging and iteration, more efficient tool use, or a combination of these capabilities.
Related posts #
- NewsPeter Hess
- [Toward fast AI interpretability and control](https://research.ibm.com/blog/fast-interpretability-plugin-vllm) NewsKim Martineau
- [llm-d 1.0 hardens open-source AI inference for day 2 production operations](https://research.ibm.com/blog/llmd-1-0-is-ready-for-production-inference-serving) ReleasePeter Hess
- A new IBM Research specialization teaches how AI models run on modern accelerators Technical noteKaoutar El Maghraoui