OpenAI says its newest models can migrate COBOL into modern languages with high accuracy, and it is true that the numbers are striking. Academics who spoke to The Verge for a report published October 26 argue the headline answers an easier question than the one banks and governments actually face. Rewriting code, one of them said, "is not the same as replacing a system."
This post lays out what OpenAI measured, what the experts say it leaves out, and how a team weighing an AI-assisted migration should test the claims. It builds on our earlier write-up of OpenAI's MultiLangBench and COBOL eval, which covered the numbers; this one covers the reaction.
TL;DR #
| Question | Answer |
|---|---|
| What did OpenAI claim? | On October 19, GPT-6.1 Sol preserved COBOL program behavior 87% of the time at low reasoning effort and 98% at higher effort |
| How was it scored? | A migration counts only if all tests pass, per OpenAI as quoted by The Verge |
| Prior models in the same test (low effort) | GPT-6 Astra about 86%, GPT-6 Sol 21%, GPT-5.6 Sol 5% |
| What do experts say? | The evals are "superficial" and skip the hard part: decomposing a legacy system and proving nothing changed |
| Does OpenAI agree? | Its spokesperson said legacy code "is still far from a solved problem" |
| What should teams do? | Treat AI as a translation and documentation aid, and keep parallel-run testing and human review |
Why COBOL still matters #
COBOL, short for common business-oriented language, dates to 1959. The Verge cites IBM as saying it is the foundation of more than 40 percent of online banking systems, supports 80 percent of in-person credit card transactions and handles 95 percent of ATM transactions. It also sits under government systems from Social Security to the IRS and many state services. One rough estimate in the piece puts the annual cost of COBOL's inefficiencies to US GDP at about $105 billion.
Replacement is hard for reasons that have little to do with syntax. The systems are too important to risk breaking, and many of the programmers who understood them have retired. Fortran has a similar profile in scientific computing, weather modelling and nuclear security code. For years AI coding tools made little headway here, largely because there is far less public COBOL to learn from than Python or JavaScript.
What OpenAI measured #
OpenAI's post, Advancing coding in low-resource programming languages, reported "substantial improvements" in migrating older languages to modern code, including C++, Java, Kotlin, Python and Rust, without changing what the software does. The Verge quotes OpenAI that "a migration is considered successful only if all tests pass."
According to The Verge's account of OpenAI's tests, GPT-6.1 Sol scored 87 percent behavior preservation at low reasoning effort and 98 percent at higher effort. At low effort GPT-6 Astra scored about 86 percent, GPT-6 Sol 21 percent and GPT-5.6 Sol 5 percent. OpenAI also reported gains in building database software across languages, including COBOL, which is the MultiLangBench side of the release. Other outlets report slightly different decimal figures (86.6 and 97.9 percent), which is consistent with rounding. For the cost angle of these models, see our piece on GPT-6.1 Sol and Astra cost efficiency and the original GPT-6 Astra launch.
Green blocks crossing a bridge from one tray to another, illustrating code migration between languages in an AI COBOL migration
One detail from OpenAI's own example is worth keeping: COBOL fixed-point arithmetic discards extra decimal digits unless rounding is requested, so a rewrite that rounds can run cleanly and still change financial results. That is why behavior preservation is the right metric, and why hidden tests that demand exact output parity are a sensible design.
What the experts told The Verge #
Four researchers were quoted, and their critiques overlap.
Vadim Zaytsev, University of Twente. "Rewriting code is not the same as replacing a system." He notes that the largest system he has seen had 250 million lines of COBOL, and that code is only part of the problem. A system may span several languages, store data in many formats, integrate with other systems and carry the expectations of the people who depend on it. His conclusion: AI is not playing a major role at the moment in replacing or migrating legacy COBOL systems. His sharpest illustration: if an AI handles 99 percent of your code, the remaining 1 percent could be a million lines "subtly failing" in ways that are hard to predict.
Arie van Deursen, Delft University of Technology. He called OpenAI's blog "superficial," saying it gave few details on the migration task and that the benchmark tested relatively simple, well-defined programs similar to what models would have seen in training. He stressed the gap between maintaining and modernizing a system, calling the tasks "lightyears" from real modernization. In his words, the evals "don't address the key difficulty": breaking a complex legacy system into smaller units that can be replaced individually.
Alastair Donaldson, Imperial College London. The main challenge is often not rewriting but understanding how the system works, a "very common" problem when few people know the language. He added that "nobody in their right mind would use COBOL for a new development project," and pointed to robust testing as the crux: run the same workflows through the old and new systems and check that they behave identically.
Jacopo Mauro, University of Southern Denmark. Formal verification, which mathematically checks that code behaves correctly, could automate that check and would beat a human review, but applying it to legacy code would take a "huge amount of work." Until then, human verification "remains essential."
OpenAI spokesperson Oscar Haines told The Verge that legacy code "is still far from a solved problem" and that the company is exploring better tests and improved model understanding of these languages. Anthropic did not comment in time for publication.
Anthropic's parallel pitch #
OpenAI is not alone. In February 2026, Anthropic published a guide to modernizing COBOL with Claude Code and argued that "the economics of COBOL modernization have shifted," according to The Verge, which notes it offered no benchmarks. Markets reacted anyway: IBM shares fell about 13 percent that day, as CNBC reporting summarized on Slashdot put it, on fears that AI would erode consulting-led modernization work at IBM, Accenture and Cognizant.
That reaction shows how much commercial weight rides on this question. For related Anthropic coverage of how Claude is being sold into enterprises, see Anthropic's embedded evaluation with Accenture.
Claims, evidence and gaps #
| Claim | Source | What is shown | What is missing |
|---|---|---|---|
| Sol reaches 87% to 98% behavior preservation | OpenAI | Scores on an OpenAI-run eval with hidden tests | Independent reproduction, program sizes and complexity |
| Newer models improved massively (5% to 87% at low effort) | OpenAI | Generational comparison | Whether the tasks resemble production code |
| Migration is the bottleneck | OpenAI framing | Eval design | Experts say understanding and decomposition are the bottleneck |
| COBOL economics have shifted | Anthropic | Guide, no benchmarks per The Verge | Measured results on real systems |
| Savings and speed are real | Experts, cautiously | Qualitative agreement | Evidence that partial automation is enough for critical systems |
The accurate reading is that models are much better at translating well-specified legacy programs, and that this is a real step. It is not evidence that a bank can switch off a mainframe.
A practical playbook for teams #
Drawing on the experts' suggestions and the structure of the evals, a cautious team could proceed like this:
- Inventory first. Map programs, data stores, batch schedules and integrations. The decomposition step is the one experts say AI evals skip.
- Pick seams. Choose small units with clear inputs and outputs, so a migrated piece can run beside the original.
- Build the oracle. Capture production-like inputs and expected outputs, including edge cases like rounding and truncation. Behavior preservation is only as good as the tests.
- Parallel run. Feed the same workflows to old and new systems and diff results, as Donaldson suggests.
- Make the model explain itself. Ask for documentation of each translation so people who know the legacy language can review it.
- Keep a human gate on anything that touches money, identity or safety. Consider formal verification for the highest-risk modules.
- Track the leftover. Zaytsev's 1 percent is where outages hide, so list unconverted or low-confidence code explicitly.
Teams already doing AI-assisted rewrites at smaller scale have comparable lessons. Our coverage of the Bun Zig-to-Rust rewrite, OpenAI's Habitat Python-to-Rust rewrite and the Turso SQLite rewrite shows what works when the original has good tests, and what is different when it does not. For the enterprise-agent side, see Marc Andreessen on Cognition's Devin and COBOL.
What this means if you build or buy #
Engineers on legacy stacks. Models can now be a serious assistant for reading, documenting and translating COBOL, and the cost of an exploratory migration has dropped. Use them to build understanding and test scaffolding as much as to produce code.
Buyers of modernization services. Ask vendors for results on your own code, not benchmark scores, and for how they prove equivalence. A 98 percent eval score is a ceiling for easy cases, not a project estimate.
Investors and executives. The IBM reaction in February showed markets pricing in fast displacement. The experts here argue the bottleneck is organizational and architectural, which moves slowly.
What to watch next #
- Independent reproductions of OpenAI's COBOL migration eval, or a public release of the migration tasks (the MultiLangBench setup is public on GitHub per OpenAI, the migration eval is described as hidden tests).
- Published case studies from banks or agencies with measured defect rates after AI-assisted migration.
- Whether Anthropic publishes benchmarks to back its February claim.
- Tooling for formal verification of translated code.
Figures are as reported by OpenAI and The Verge on October 26, 2026, and may be revised.