IBM zEnterprise EC12 and BC12 mainframes on display, the kind of machines that still run much of the world's COBOL. Illustrative. Image: Agiorgio / Wikimedia Commons, CC BY-SA 4.0, cropped
OpenAI says its newest models have gone from almost useless to near-perfect at translating COBOL, the 1950s-era language that still sits underneath much of the world's banking. In a research post published on October 19, the company reported that GPT-6.1 Sol correctly migrated 97.9% of COBOL programs in its own test, against 5.4% for last generation's GPT-5.6 Sol.
The post is about what OpenAI calls low-resource programming languages: COBOL, Fortran, Visual Basic and others with far less public code and documentation for models to learn from than Python or JavaScript. OpenAI says developers and enterprise customers kept telling it that coding models struggle with them.
Why old code matters #
These languages are old, but they are not niche. OpenAI cites IBM figures that COBOL supports more than 40% of online banking systems, 80% of in-person card transactions and 95% of ATM transactions. Fortran runs weather forecasting and nuclear reactor safety analysis. The people who wrote much of that code have often retired, documentation is patchy, and key business rules may exist only in the code itself.
That makes rewriting it risky. OpenAI gives an example: COBOL's fixed-point arithmetic throws away extra decimal digits unless rounding is explicitly asked for, so a modern rewrite that rounds instead can run without a single error while quietly producing different financial results.
From 5.4% to 97.9% on COBOL migration #
To test this, OpenAI built a COBOL migration evaluation that asks a model to convert COBOL programs into C++, Java, Kotlin, Python or Rust. Hidden tests demand byte-for-byte identical output, and a migration only counts if every test passes.
At low reasoning effort, GPT-6.1 Sol, the cheaper model OpenAI unveiled at DevDay, scored 86.6%, compared with 5.4% for GPT-5.6 Sol at the same setting, at 61% of the API cost. At higher effort it reached 97.9%, which OpenAI says is close to saturating the test. The company adds that production code still needs expert review and testing against a system's business rules before anyone ships a migration.
Building SQLite in 20 languages #
The second test, MultiLangBench, asks a model to build a SQLite-compatible database engine from scratch in one of 20 languages over a long coding session, then checks it against tests adapted from SQLite's own suite. OpenAI is releasing the benchmark on GitHub with per-language results.
At maximum reasoning effort, flagship GPT-6 Astra averaged an 87.4% pass rate, up from 26.8% for GPT-5.6 Sol, and improved in every language. It scored above 83% in 19 of the 20. The gains in the older languages were the biggest:
- COBOL: 10.9% to 72.1%, still the hardest language in the set
- **Fortran:** 21.4% to 83.5%
- **Visual Basic:** 27.5% to 89.4%
- **OCaml:** 25.6% to 85.8%
The budget model, GPT-6 Luna, managed a 40.3% average, beating GPT-5.6 Sol in every language at an estimated $0.77 per sample against $3.48, about 78% cheaper. Its COBOL score rose only from 10.9% to 17.1%.
Who can use it #
All three models are available through OpenAI's API, and the company is inviting developers to try them on their own languages and report where they fall short. Both evaluations are OpenAI's own, and the COBOL migration test has not been published, so the figures have not been independently checked. OpenAI also does not compare its models with rivals such as Anthropic's Claude, which many developers use for coding; Replit recently switched from Claude to OpenAI.
Why it matters #
Banks, insurers and governments have spent decades trying, and often failing, to move off COBOL because rewrites are slow, costly and easy to get subtly wrong. If OpenAI's numbers hold up outside its own tests, AI could make those migrations far cheaper, which also makes legacy modernisation a serious enterprise market for OpenAI.
Sources: OpenAI, MultiLangBench on GitHub.