OpenAI’s COBOL migration score has made a striking leap
The New Stack Meredith Shubel
OpenAI says its newest models are way better at moving COBOL code. The jump looks huge, but the company isn’t showing the full test data.
Based on reporting by The New Stack, Meredith Shubel — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
OpenAI just put COBOL on the front page of its model pitch. The company says its latest systems are much better at migrating old COBOL programs and at building software in languages where there’s less code and documentation to learn from.
The headline number is hard to ignore. On OpenAI’s own COBOL migration test, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort, up from GPT-5.6 Sol’s 5.4%, while using 39% less. At higher reasoning effort, it hit 97.9%.
But the test itself is only partly visible. OpenAI has not published the details of its COBOL evaluation. It says the benchmark checks whether a model can move existing programs to C++, Java, Kotlin, Python, or Rust while preserving behavior, and that success requires hidden tests with byte-for-byte parity on observable outputs. The company has released MultiLangBench on GitHub, though, which asks models to build a SQLite-compatible database engine across 20 programming languages.
That second benchmark also shows large gains. GPT-6 Astra averaged an 87.4% pass rate across the 20 languages, up from GPT-5.6 Sol’s 26.8%, and scored above 83% on 19 of them. GPT-6 Luna came in lower at 40.3%, but still beat GPT-5.6 Sol in every language while carrying an estimated API cost about 78% lower.
Still, benchmarks are not the same as production code. OpenAI itself says using this approach on real software still needs expert review and testing against business rules and dependencies. And because the COBOL test cases are hidden, outside developers can’t inspect the setup or verify the claim for themselves. That matters, especially when COBOL still underpins a huge share of banking and transaction systems, and when rivals like Anthropic and AWS have already been circling the same modernization problem.
My take — AI-written commentary, not fact-checked reporting
The big tell here is not the score; it’s the missing test suite. OpenAI wants the applause for progress, but not the receipts, which is a very familiar move in AI land. If the claims are solid, publish the cases and let people kick the tires.
Read more about this at: The New Stack