# Why Your COBOL Migration Should Keep the Bugs > Source: > Published: 2026-08-03 18:08:25+00:00 [AI](https://sourcefeed.dev/c/ai)Article # Why Your COBOL Migration Should Keep the Bugs American Express researchers show where LLMs actually belong in mainframe modernization: writing the tests, not the Java. [Mariana Souza](https://sourcefeed.dev/u/mariana_souza) A paper that hit Hacker News under the headline "AI migrated legacy COBOL programs to Java, bugs included" sounds like a cautionary tale about sloppy LLM output. Read the actual paper and it's the opposite: the bugs are the point. A team led by American Express distinguished engineer Andras Ferenczi built a system whose explicit goal is to prove the migrated Java misbehaves *exactly* the way the original COBOL does — and the architecture they used says a lot about where LLMs do and don't belong in legacy modernization. That inversion deserves attention, because most of the industry is still selling the other approach. ## Bug-for-bug is the spec If you've never worked near a mainframe, "preserve the bugs" sounds insane. If you have, it's obvious. A COBOL program that's been clearing card transactions since 1987 doesn't have a spec — it *is* the spec. Every quirk, every weird rounding decision, every special case someone patched in decades ago has downstream consumers that depend on it. That's Hyrum's Law operating at maximum intensity: with enough users over enough years, every observable behavior of your system becomes load-bearing, whether it was intended or not. So "fix the bugs during migration" is a category error. Fix a bug and you've silently changed reconciliation output for some batch job three systems away, and you'll find out at month-end close. Serious migration shops have always aimed for bug-for-bug compatibility first and cleanup second. The hard part was never wanting parity — it was *proving* it, on code with no test suite, no test data, and no one alive who remembers the edge cases. That proof gap is what the [arXiv paper](https://arxiv.org/abs/2607.28271) attacks. ## The interesting part: the LLM never writes the Java Here's the architectural choice that separates this from the "paste your COBOL into a chatbot" genre. In the team's "Locksmith Loop," a deterministic, rule-based migrator does the actual COBOL-to-Java translation — an AST-to-AST transformation, no model in the loop. The AI agent's job is adversarial: it hunts for inputs and program paths where the two versions might diverge. The loop runs both the COBOL and the generated Java off-mainframe, instrumented with mocks, and lets an agent explore the input space using a battery of search strategies — pairwise and three-way interaction testing, Latin hypercube sampling, adaptive random testing, MAP-Elites, UCB1 bandit selection. When input search plateaus, the agent proposes "parity-preserving mutations": symmetric code edits applied to *both* sides that expose hard-to-reach paragraphs without changing observable behavior. Paragraphs the search can't reach get flagged as "Locked" and worked on explicitly. Every agent proposal is accepted or rejected by deterministic execution — the model gets no vote on correctness. And when a divergence turns up? They fix the *migrator*, not the generated file. That's the detail worth stealing. A hand-patched output file is a one-off; a fixed transpiler rule makes the same defect impossible across every program you migrate afterward. The numbers are solid for what they are: 96.8% branch coverage on a 430-line program from [AWS's CardDemo](https://github.com/aws-samples/aws-mainframe-modernization-carddemo) sample app, 91.9% on a 4,114-line production-grade program with 142 paragraphs, where 166 generated test cases passed parity with zero divergences, no human in the loop. ## Where the skepticism lands — and where it misses The HN thread (77 comments, heavy mainframe-veteran representation) raised the right objections. A 4,114-line program is a toy next to real estates that run to millions of lines across CICS transactions, JCL job streams, and VSAM files — none of which this method touches. The authors themselves list the caveats: three programs total, branch coverage is an imperfect proxy for fault detection, and the parity gate is silent about intermediate state between checkpoints. Fair, all of it. But one popular criticism — "the LLM didn't even do the migration!" — misreads what's valuable here. The pure-LLM translation papers this contrasts with report things like "93% accuracy," and 93% is a horrifying number when the domain is money. Deterministic transpilation plus agentic *validation* is the only division of labor that survives contact with an auditor. The LLM contributes what it's genuinely good at — creative search, hypothesis generation, harness authoring — inside a system where a wrong guess costs nothing because execution is the oracle. The honest criticism is different: deterministic transpilers emit what practitioners derisively call JOBOL — Java with COBOL bones, paragraph-shaped methods and all. [AWS Transform](https://aws.amazon.com/transform/) (built on the Blu Age engine) has fielded this complaint for years, and [IBM's watsonx Code Assistant for Z](https://www.ibm.com/products/watsonx-code-assistant-z) leans on generative AI partly to produce more idiomatic output. There's a real tension here: the more idiomatic the Java, the harder parity is to prove; the more provable, the less maintainable. This paper plants its flag firmly on the provable end and doesn't pretend otherwise. ## What to actually do with this If your team is eyeing an AI-assisted migration, the transferable lessons are concrete: **Don't let a model generate production code for a system with no test suite.** Use the model to*build*the test suite. The original system is a runnable oracle — that's an asset most greenfield projects would kill for.**Get both versions running off-mainframe early**, even with crude mocks. Everything in this method depends on cheap, repeatable side-by-side execution. If your COBOL only runs on the LPAR, you can't iterate.**Measure branch coverage on the COBOL side**, not test count on the Java side. "Our generated tests pass" means nothing if they exercise 40% of the branches.** Route every fix through the translation layer.**If you're hand-editing generated output, you've forked it, and your parity evidence is now stale. My read: this is the most credible validation story anyone has published for AI-era mainframe migration, and it's still maybe a third of the problem. Data migration, batch orchestration, performance on decimal-heavy workloads, and the JOBOL maintainability debt all sit outside its scope. But it moves the argument from "trust the model" to "check the execution," and for the banks and card networks that still run the world's transactions, that's the only argument that was ever going to work. Expect the vendors to bolt something Locksmith-shaped onto their pipelines within a year — the transpilers were never the hard part. ## Sources & further reading - [Agentic Method for Deterministic Validation of Legacy Code Migration](https://arxiv.org/abs/2607.28271)— arxiv.org - [AI migrated legacy COBOL programs to Java, bugs included](https://news.ycombinator.com/item?id=49150773)— news.ycombinator.com - [AWS Mainframe Modernization CardDemo](https://github.com/aws-samples/aws-mainframe-modernization-carddemo)— github.com - [IBM watsonx Code Assistant for Z](https://www.ibm.com/products/watsonx-code-assistant-z)— ibm.com [Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. ## Discussion 0 No comments yet Be the first to weigh in.