{"slug": "ai-cobol-migration-what-openai-s-new-coding-results-show", "title": "AI COBOL Migration: What OpenAI's New Coding Results Show", "summary": "OpenAI published results on October 19, 2026 showing GPT-6 Astra averaging an 87.4% test pass rate across 20 low-resource programming languages on its MultiLangBench build-SQLite-from-scratch benchmark, up from 26.8% for GPT-5.6 Sol, while GPT-6.1 Sol reached 97.9% on OpenAI's COBOL-to-modern-language migration test at higher reasoning effort versus 86.6% at low effort. COBOL remained the hardest language at 72.1%, up from 10.9%, and OpenAI says GPT-6.1 Sol matched the migration task at 61% of the API cost of GPT-5.6 Sol, which scored 5.4% at the same low setting. Both evaluations were built and run by OpenAI, the migration test is not public, and OpenAI's post states that production use still requires expert review and testing against a system's business rules and dependencies.", "body_md": "On October 19, 2026 OpenAI published results for its models on what it calls low-resource programming languages: COBOL, Fortran, Visual Basic and others with little code or documentation in public training data. The headline numbers are an 87.4% average test pass rate for GPT-6 Astra on a build-SQLite-from-scratch benchmark across 20 languages, and 97.9% for GPT-6.1 Sol on a COBOL-to-modern-language migration test at higher reasoning effort. Both evaluations are OpenAI's own, and both were run by OpenAI.\n\nThe reason this matters to a business reader is the installed base. OpenAI's post, citing IBM, says COBOL sits under more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions. Fortran still runs weather and nuclear safety codes. The people who wrote these programs are often gone, the documentation is partial, and the business rules live only in the code. That is exactly the kind of estate where a migration either preserves behaviour exactly or quietly produces different financial results.\n\nSo the question for a CIO or engineering lead is not whether the scores are impressive. They are. The question is what a score on an OpenAI-built test tells you about your own code, and how to find out without betting a system on it. We read the post and the public benchmark repository on October 21, 2026 and set out what was measured, what was not, and how to run a pilot.\n\n1. 01Two tests, two different skills.MultiLangBench measures building new software: a SQLite-compatible engine, in a requested language, from scratch. The COBOL migration evaluation measures preserving existing behaviour while changing language. OpenAI says they are complementary, and the results should be read separately.\n2. 02GPT-6 Astra averages 87.4% across 20 languages, up from 26.8%.OpenAI reports gains in every language tested, with Astra above 83% in 19 of 20. COBOL went from 10.9% to 72.1% and stays the hardest. Fortran reached 83.5%, Visual Basic 89.4% and OCaml 85.8%.\n3. 03COBOL migration reached 86.6% at low effort and 97.9% at higher effort.GPT-6.1 Sol migrated COBOL programs to C++, Java, Kotlin, Python or Rust against hidden tests that demand byte-for-byte parity on observable outputs. GPT-5.6 Sol scored 5.4% at the same low setting, and OpenAI says the new model did it at 61% of the API cost.\n4. 04All of it is OpenAI measuring OpenAI.Both evaluations were built and run by OpenAI. MultiLangBench is published on GitHub under Apache 2.0 so you can inspect what it tests; the migration test is not public. No rival lab has published a comparable number, so there is no cross-vendor comparison to make.\n5. 05OpenAI's own caveat is the operating rule.The post states that applying this to production software still requires expert review and testing against the system's business rules and dependencies. A pilot on your own programs, scored by your own tests, is the only number that transfers.\n\n## 01 — The NumbersWhat OpenAI *measured*, and against what baseline.\n\nThe post compares current models against GPT-5.6 Sol, OpenAI's own earlier model, on two evaluations it built for the purpose. That framing matters. Every improvement is a gap between two OpenAI models on an OpenAI test, not a comparison with a human team, a migration vendor or another lab's model. Read that way, the numbers are clean and worth having.\n\nThe four figures below are the ones OpenAI leads with. Each is stated as reported. The cost figure for GPT-6 Luna is an estimate of API spend per sample; the cost figure for the migration test is stated as a share of what GPT-5.6 Sol spent at the same setting. We explain the two cost claims in section 05, because the post gives the migration saving two ways.\n\n##### GPT-6 Astra, average test pass rate\n\nAcross 20 languages at maximum reasoning effort. GPT-5.6 Sol: 26.8%. Gains in every language tested.\n\n##### GPT-6 Luna, average test pass rate\n\nThe cheaper model also beat GPT-5.6 Sol in every language, at an estimated $0.77 per sample against $3.48.\n\n##### GPT-6.1 Sol at low reasoning effort\n\nGPT-5.6 Sol scored 5.4% at the same setting. OpenAI says the new model spent 61% of the API cost.\n\n##### GPT-6.1 Sol at higher reasoning effort\n\nOpenAI describes this as nearly saturating the evaluation. A saturated test stops discriminating between models.\n\nOne example of why the baseline matters: a 5.4% to 86.6% jump on migration reads like a sixteen-fold improvement, and arithmetically it is. But a 5.4% baseline also says the earlier model was close to useless on this specific test, so most of the gain is the test becoming passable at all. The failure case is quoting the ratio in a board paper as a measure of how much better migration has become in general. It is a measure of how much better one model got on one test.\n\n## 02 — Benchmark OneBuilding SQLite from scratch in *twenty languages*.\n\nMultiLangBench is the public half of the work. The task is to implement a SQLite-compatible database engine in a requested language, over an extended coding session, with the model expected to organise a codebase, manage state, handle errors and add features without breaking earlier ones. The repository on GitHub, released under Apache 2.0, states that the model is given the SQLite documentation and no internet access, and that every trace of SQLite itself is removed from the container. A solution written in the wrong language, or in a mix of languages, scores zero.\n\nScoring uses database files as the common interface. The candidate engine receives a starting database and a SQL query and must produce a modified database. The result is checked against what real SQLite 3.40.1 produces. OpenAI adapted the tests from SQLite's own suite; the repository says about 24,000 of roughly 51,000 TCL tests could be converted, and recommends scoring on a fixed subset of 272 tests with a 60-second limit per test. A crash, a mismatch or a timeout each count as a failure.\n\nThe per-language results are where the post is most useful, because they show which languages remain hard. OpenAI reports GPT-6 Astra above 83% in 19 of 20 languages, with COBOL the outlier.\n\n#### MultiLangBench test pass rate, GPT-6 Astra, four named languages\n\nSource: OpenAI, 'Advancing coding in low-resource programming languages', October 19, 2026. OpenAI-run, maximum reasoning effort. The post names these four languages and says per-language results accompany the GitHub release; the repository checked October 21 held the test suite and setup notes but no per-language scores.\nWhy COBOL lags is not stated in the post, but the task hints at it. Building a database engine means managing memory, strings and recursion in a language designed for fixed-format business records, with little idiomatic open-source code to learn from. The example to hold onto is Fortran: it has the same scarcity problem and reached 83.5%, so scarcity alone does not explain the gap. The failure case for a reader is treating 72.1% as a verdict on COBOL migration. It is a score for writing a new COBOL program of unusual shape, which is not the job most estates need done.\n\n## 03 — Benchmark TwoThe COBOL migration test and its *byte-for-byte* rule.\n\nThe second evaluation is the one a legacy estate cares about. OpenAI asked its models to migrate COBOL programs to one of five targets: C++, Java, Kotlin, Python or Rust. Success is judged on hidden tests that require byte-for-byte parity on observable outputs, and a migration counts as successful only if every test passes. That is a strict rule, and it is the right one for financial code, where a rewrite that compiles and runs can still be wrong.\n\nThe results, as OpenAI reports them, are in the table. The post does not say how many programs were migrated, how long they were, or how they were selected. It does not publish the test. Those gaps are what keep this a vendor result rather than a reproducible one.\n\n| Source: OpenAI, October 19, 2026. OpenAI-run on a private evaluation; targets C++, Java, Kotlin, Python or Rust; hidden tests requiring byte-for-byte output parity. Read October 21, 2026. |  |  |  | \n|---|---|---|---|\n| Model and setting | Accuracy | Cost, as stated | What it means | \n|---|---|---|---|\n| GPT-5.6 Sol, low effort | 5.4% | Baseline | The earlier model rarely passed every hidden test on a program | \n| GPT-6.1 Sol, low effort | 86.6% | 61% of the baseline's API cost | Roughly seven in eight programs migrated with every output byte matching | \n| GPT-6.1 Sol, higher effort | 97.9% | Not stated | OpenAI calls the test close to saturation at this setting | \n\nThe example worth copying is the pass rule, not the score. If your own pilot accepts a migration when most tests pass, you will get a higher number than OpenAI's and a worse system. The failure case is the quiet one: a migrated batch job that runs to completion and posts figures a few cents different from the original, discovered at month end. All-tests-pass is the only standard that catches it.\n\n## 04 — BehaviourWhy preserving *behaviour* is the hard part.\n\nOpenAI's post gives one example of what a migration has to get right, and it is a good one. COBOL's fixed-point arithmetic throws away extra decimal digits unless the program asks for rounding. A rewrite in a language whose numeric types round by default will compile, run and produce different totals. Nothing crashes. The books are simply wrong.\n\nThat is one of many such rules. Missing-record handling, date formats, field padding, sign conventions and the exact order in which a batch processes its input all encode business decisions that nobody wrote down. A model that understands COBOL as a language is not enough; it has to reproduce the program as it behaves, which is why OpenAI's test checks outputs rather than code.\n\nThe practical consequence is that test coverage, not model choice, decides whether a migration can be trusted. OpenAI could score 97.9% because it had hidden tests that pin every observable output. Most estates do not. Building that harness is the first job of a pilot and usually the longest, and it is the part no model does for you. Our [order-of-operations guide for AI transformation](https://www.digitalapplied.com/blog/ai-transformation-sequence-smb-order-of-operations) puts measurement before automation for the same reason.\n\n## 05 — CostWhat one attempt *costs*, and the two ways OpenAI states it.\n\nOpenAI gives cost in two forms and they are easy to confuse. For MultiLangBench it reports an estimated API cost per sample: $0.77 for GPT-6 Luna against $3.48 for GPT-5.6 Sol, which it rounds to approximately 78% lower. That is a per-attempt figure for building an engine, and Luna reached only 40.3% average pass rate at that price, so it is a cheaper partial result, not a cheaper equivalent one.\n\nFor the migration test the post says GPT-6.1 Sol at low effort cost 39% less than GPT-5.6 Sol in its summary and, in the detailed section, that it ran at just 61% of the API cost. Those are the same claim stated two ways. No absolute dollar figure is given for the migration test, and nothing is stated for the higher-effort run that reached 97.9%, which will have cost more per attempt.\n\n##### GPT-6.1 Sol, higher effort\n\nThe number to quote for capability. Cost at this setting is not stated by OpenAI, so budget from your own pilot, not from the post.\n\n##### GPT-6.1 Sol, low effort\n\nCheaper and still far ahead of the old model. The trade is one program in eight failing the all-tests-pass rule and needing a second attempt or a human.\n\n##### GPT-6 Luna\n\nBuilds partial engines for a fifth of the price. Fine for exploration and tooling, not the model to migrate a ledger with.\n\nThe example to run in your own spreadsheet: if a program takes two attempts at low effort for every one at higher effort, the cheaper setting may cost more in total once review time is counted. The failure case is choosing the setting by price per call and ignoring the price of a failed migration that a human then has to debug. Current list prices for these models are in our [GPT-6.1 Sol pricing and upgrade guide](https://www.digitalapplied.com/blog/gpt-6-1-sol-pricing-benchmarks-upgrade-guide) and our [GPT-6 Astra price and benchmarks guide](https://www.digitalapplied.com/blog/gpt-6-astra-price-benchmarks-guide).\n\n## 06 — FencesWhat these results *do not* say.\n\nFour things are missing from the post, and each one changes how the numbers should be used. First, there is no comparison with any other lab's model. Anthropic, Google and the open-weight labs have not published results on these tests, and OpenAI did not run them, so \"best model for COBOL\" is not a claim the post supports. Second, the migration evaluation is private: its programs, sizes and selection method are unpublished, so nobody outside OpenAI can reproduce the 97.9%.\n\nThird, the sample is unknown. A test built from short, self-contained COBOL programs with clean inputs will score differently from one built on a 300,000-line batch system with CICS calls and copybooks shared across a dozen jobs. The post does not say which it used. Fourth, the task ends at parity. Nothing in either evaluation measures whether the migrated code is maintainable, idiomatic, or faster, and those are the reasons most organisations migrate in the first place.\n\nOpenAI's own sentence on this is the one to keep. The post says that applying these capabilities to production software \"still requires expert review and testing against the system's business rules and dependencies.\" That is not a disclaimer; it is the project plan. The example is a bank that migrates a rate-calculation module at 97.9% parity and discovers the remaining 2.1% is the leap-year handling. The failure case is a programme that budgets for the 97.9 and not the 2.1.\n\n## 07 — PilotHow to run a pilot on *your own* code.\n\nThe only migration score that transfers is one measured on your programs with your tests. OpenAI's method is a good template: pick programs, capture their observable outputs on real inputs, migrate, and accept only when every output matches byte for byte. The sequence below is how we would scope four to six weeks, with the model as the last decision rather than the first.\n\nOne example of a result that changes the decision: if three of your ten programs fail on the harness itself because production inputs are not reproducible, the finding is that the estate is not ready to migrate by any method, and that is worth knowing before a vendor quotes. The failure case is the opposite: a pilot on three hand-picked programs that all pass, followed by a contract for the other 1,200. Our [AI transformation practice](https://www.digitalapplied.com/services/ai-transformation) runs the harness-first version of this with clients, and the broader model context sits in our [GPT-6 Sol and Luna launch guide](https://www.digitalapplied.com/blog/gpt-6-sol-luna-launch-pricing-benchmarks-2026).\n\n## 08 — ConclusionA vendor's *97.9%* is the start of a pilot, not the end of one.\n\n### Build the output harness for ten of your own programs, score the model by all-tests-pass, and budget for the programs that fail.\n\nOpenAI's October 19 results are real progress on a real problem. Building a SQLite-compatible engine in COBOL went from 10.9% to 72.1% between two of its own models, and migrating COBOL programs with every output byte matching went from 5.4% to 86.6% at the same effort setting, and 97.9% with more. Those are OpenAI's numbers on OpenAI's tests, one of which is public and one of which is not, and no other lab has published a comparison.\n\nFor an organisation holding COBOL, Fortran or Visual Basic, the right reading is that AI-assisted migration is now credible enough to pilot, and that the pilot's value lies in the harness and the review, not the model. Capture what your programs do, migrate a representative sample, accept nothing short of full parity, and treat every remaining difference as a business rule to be understood. The model will get the easy 90%. Your estate is defined by the rest.", "url": "https://wpnews.pro/news/ai-cobol-migration-what-openai-s-new-coding-results-show", "canonical_source": "https://www.digitalapplied.com/blog/openai-cobol-migration-low-resource-languages-results", "published_at": "2026-10-20 00:00:00+00:00", "updated_at": "2026-10-23 00:09:52.217318+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "developer-tools"], "entities": ["OpenAI", "GPT-6 Astra", "GPT-6.1 Sol", "GPT-5.6 Sol", "GPT-6 Luna", "MultiLangBench", "IBM", "COBOL"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-cobol-migration-what-openai-s-new-coding-results-show", "markdown": "https://wpnews.pro/news/ai-cobol-migration-what-openai-s-new-coding-results-show.md", "text": "https://wpnews.pro/news/ai-cobol-migration-what-openai-s-new-coding-results-show.txt", "jsonld": "https://wpnews.pro/news/ai-cobol-migration-what-openai-s-new-coding-results-show.jsonld"}}