ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration
Hugging Face 3 months ago 10
Researchers released ScarfBench, an open benchmark for evaluating AI agents on Java framework migration tasks across Spring, Jakarta EE, and Quarkus. The benchmark contains 34 applications, 204 migration tasks, and approximately 151,000 lines of code, with success measured by whether applications build, deploy, and preserve behavior. Current frontier AI agents achieve less than 10% behavioral success on the benchmark, revealing that dependency management across configuration, infrastructure, and runtime environments—rather than code translation itself—is the primary migration challenge.