When OpenAI published benchmark results for COBOL on October 19, GPT-6.1 Sol achieved 97.9% accuracy on a byte-for-byte behavioral fidelity evaluation for COBOL migration — a 16-fold improvement over the 5.4% its predecessor managed at the same test. That figure represents a genuine capability threshold crossing: the AI technical barrier to accurate COBOL translation has moved meaningfully. What hasn't moved are the copybooks, the JCL job streams, the undocumented business logic, and the organizational complexity that make real enterprise migration hard — none of which that benchmark measures.

The distinction matters because an estimated 250 billion lines of COBOL remain in active production globally. IBM's own statistics show that COBOL runs 95% of ATM transactions globally, 80% of in-person credit card transactions, and more than 40% of all online banking systems worldwide.

The workforce maintaining those systems is aging: the average COBOL developer was 58 years old as of a 2019 workforce study, with approximately 10% retiring annually, and 71% of mainframe teams report understaffing, per a Forrester survey commissioned by Deloitte. The case for AI assistance is urgent. What GPT-6's numbers actually demonstrate — and what they leave open — is what enterprise IT teams need to understand before restructuring their modernization budgets around it.

A Problem Measured in Billions of Lines

COBOL — Common Business-Oriented Language — was first standardized in 1960, developed under US Department of Defense sponsorship and designed specifically for business data processing — a fixed-field language with prose-like syntax, rigid four-division program structure, and more than 300 reserved words. Its distinguishing characteristics include fixed-point arithmetic with implicit decimal precision, numeric and alphanumeric type coercion that converts data silently, and a level-number data declaration system that defines data layouts with a specificity modern languages handle through type systems.

Those characteristics are precisely what make it hard to translate accurately. A COBOL program that calculates a pension payout may embed rounding rules that differ from the behavior of Python's decimal module by a fraction of a cent per calculation — a difference that becomes material across millions of transactions and constitutes a regulatory compliance failure. The behavioral fidelity standard that OpenAI set for its COBOL migration evaluation directly reflects this reality: a migration counts as successful only if the output code passes all hidden tests, including tests specifically targeting edge cases in rounding, missing-record handling, and COBOL-specific numeric coercions.

Fortran — developed at IBM in the 1950s — faces a parallel problem in scientific computing. It supports weather forecasting models, nuclear reactor safety analysis, and computational fluid dynamics at facilities including NOAA and the European Centre for Medium-Range Weather Forecasts. OpenAI's LRPL announcement covers Fortran applications in detail. Fortran code running in these environments has often been maintained by the same researchers who wrote it for decades; migration means transferring both the code and the institutional understanding of what the code is actually doing.

The enterprise modernization imperative is not theoretical. Gartner's June 2026 mainframe forecast predicted that more than 70% of mainframe exit projects initiated in 2026 will fail to produce their intended benefits, citing overestimation of generative AI tooling as the primary cause. Against that backdrop, the ACT-IAC Legacy Code Modernization report estimates 84,000 unfilled mainframe positions in the United States alone.

What MultiLangBench Measures — and Why It's Designed That Way

OpenAI's MultiLangBench is built around a single specification task: construct a SQLite-compatible database engine from scratch in the specified programming language. The model must organize a multi-file codebase, manage program state, handle error conditions, and implement additional features without breaking existing tests — a representative mix of real-world software engineering tasks.

Crucially, the evaluation is language-agnostic: all 20 implementations are tested by feeding each program an initial database and a SQL query, then checking whether the output database matches what the reference SQLite implementation produces. This allows the same test suite — adapted from SQLite's own tests — to be applied uniformly across C, COBOL, Go, OCaml, Fortran, Zig, and the other 14 languages without needing language-specific test harnesses. OpenAI published full MultiLangBench results on GitHub for independent verification and community use.

The results reflect both the scale of GPT-6's improvement and the uneven ceiling across languages. At maximum reasoning effort, GPT-6 Astra achieves above 83% in 19 of the 20 languages tested. COBOL remains the hardest, improving from 10.9% to 72.1%. OCaml rose from 25.6% to 85.8%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%. The 20-language set also includes C, C#, C++, Common Lisp, Java, JavaScript, Lean, PHP, PowerShell, Python, Ruby, Rust, Scala, Swift, and Zig.

OpenAI also tested GPT-6 Luna, its cost-accessible option, which reaches 40.3% on average — an improvement on GPT-5.6 Sol in every language tested. COBOL improved from 10.9% to 17.1%; OCaml from 25.6% to 34.1%; Fortran from 21.4% to 31.4%. Crucially, Luna achieves those gains at an estimated API cost of $0.77 per sample, versus $3.48 for GPT-5.6 Sol — approximately 78% cheaper. For cost-sensitive enterprise workloads running large volumes of code analysis or test generation, that efficiency improvement may matter as much as the accuracy gain.

How Does GPT-6 Achieve Near-Saturation on COBOL Migration?

The COBOL migration evaluation sets a more demanding standard than MultiLangBench. A migration counts as successful only if it passes all hidden tests — the same tests used for the reference COBOL implementation — applied to the rewritten code in C++, Java, Kotlin, Python, or Rust. The candidate must produce exactly the same behavior, including the edge cases in rounding, missing-record handling, and numeric coercion that distinguish COBOL's semantics from those of modern languages.

GPT-6.1 Sol achieves 86.6% at low reasoning effort on this evaluation, compared with 5.4% for GPT-5.6 Sol at the same setting — while costing 39% less per API call. At maximum reasoning effort, accuracy reaches 97.9% on COBOL, which OpenAI describes as nearly saturating the evaluation.

The mechanism behind the improvement is not fully disclosed, but research into transfer learning for low-resource languages provides context. OpenAI's characterization of COBOL, Fortran, OCaml, and Visual Basic as "low-resource" reflects a genuine data distribution problem: models learn to code primarily from the Python, JavaScript, TypeScript, and Java code that dominates open-source repositories. The MultiPL-T transfer learning paper — which filters Python code to generate test suites, then translates to lower-resource languages and validates — demonstrated that fine-tuning on high-resource language data improves performance on low-resource languages, but that the improvement decreases for languages syntactically distant from Python. COBOL, with its CODASYL origins and fixed-field format, is among the most syntactically distant from modern languages. GPT-6's gains — particularly on COBOL — suggest either improved cross-lingual transfer mechanisms or synthetic data generation strategies not detailed in the announcement.

What the Benchmark Doesn't Measure — and Why That Matters for Enterprise Migration

The 97.9% figure is specific to what the benchmark tests: building a new program from scratch that meets a specification. The most technically demanding aspects of real COBOL enterprise migration are not in this evaluation, and understanding that gap is essential for enterprise IT teams evaluating what these results mean for their modernization pipelines.

Copybooks are shared data layout definitions that COBOL programs reference via COPY statements. A single copybook defines the data structure that dozens or hundreds of programs share — a change to one copybook propagates effects through every program that uses it. The COBOL migration evaluation tests single-file programs. Copybooks are not covered.

JCL (Job Control Language) governs how COBOL programs actually run on IBM mainframes — defining job submission, file allocation, dataset management, and program sequencing. COBOL programs don't execute standalone; they execute within JCL job streams. A translated program that passes all unit tests may still fail when integrated into the JCL workflow it belongs to.

CICS (Customer Information Control System) manages interactive transaction processing — the environment for most customer-facing banking and government COBOL programs. Migrating a CICS transaction program requires translating not just the application logic but the transaction management semantics as well.

Embedded business logic is the deepest problem: COBOL programs encode decades of accumulated business rules that are often not documented anywhere except in the code itself. The original developers may be retired. The rules may have been modified in place over 30 years without contemporaneous documentation. Factory.ai's Legacy-Bench benchmark — which tested real COBOL tasks including bug fixing in multi-program codebases — found that models produced silent wrong-output failures: a payroll calculation passing 24 of 29 tests while computing incorrect deductions, with no error or compilation failure to signal the problem.

Thirty-one of the 44 tasks no model solved across their evaluation were COBOL tasks. That benchmark predates GPT-6; it is likely GPT-6 would perform better on its tasks. But the architectural challenges — copybook linkage, behavioral discovery, silent failure modes — are not ones that benchmark fidelity alone resolves.

Research published in April 2026 on the COBOL-Coder domain-adapted model — built on Qwen2.5-Coder and fine-tuned on COBOL — explicitly noted that its evaluation "is limited to single-file programs, so cross-file dependencies, copybooks, and JCL are not covered." That limitation applies equally to GPT-6's migration evaluation. The two evaluations are measuring real and important capability — but they are measuring the cleanest version of the problem.

OpenAI acknowledged in its announcement that strong evaluation performance does not eliminate the need for expert oversight in production migration: applying these capabilities to live systems still requires review against the full context of business rules and system dependencies.

A Crowded Field

OpenAI is entering a competitive space where the incumbent has structural advantages it cannot replicate from an API. IBM's Watsonx Code Assistant for Z — now in version 2.8 following a March 2026 update — is purpose-built for COBOL-to-Java migration and integrates into IBM's mainframe ecosystem through an agentic workflow that connects directly to job control metadata, copybook libraries, and VSAM data definitions via the Model Context Protocol. IBM's enterprise distribution, its mainframe service relationships, and its access to the actual production context — the copybooks, the JCL, the job streams — give it capabilities that a foundation model operating on isolated file inputs cannot match.

Amazon's AWS Transform for mainframe — launched in 2025 — positions itself as the first agentic AI service for end-to-end mainframe modernization, with Transamerica, Allianz, and Marriott Hotels listed as referenced enterprise clients. GitHub Copilot includes COBOL explanation, test generation for undocumented functions, and refactoring suggestions as documented capabilities. Anthropic's Claude Code COBOL modernization post, published in February 2026, argued that AI changes the economics of the discovery and analysis phases — not the translation accuracy specifically, but the workflow economics that typically make enterprise migration prohibitively expensive.

Cognition, the company behind Devin, published a case study in April 2026 detailing COBOL modernization work including a refactoring project at Itaú Unibanco — Latin America's largest private bank. Cognition reported 5–6x speed gains and zero production errors on that refactoring work. It also stated explicitly that autonomous migration of transactional systems is "out of reach for agents today," limiting its own claimed capabilities to batch workload migration.

Against that backdrop, GPT-6's open release of MultiLangBench as a public evaluation is significant. If other vendors choose to report against it, it could become the standard measure of progress in this category — the way SWE-bench became the reference point for general software engineering capability. OpenAI's decision to publish it openly rather than keep it proprietary suggests a calculation that establishing a shared industry standard serves both credibility and research velocity.

Is Real AI-Assisted COBOL Migration Closer?

The trajectory these results imply is meaningful, with a specific caveat. The gap from 5.4% to 97.9% on behavioral fidelity is not incremental — it represents a qualitative change in what the AI can do on the cleanest version of the translation problem. For enterprises with straightforward, well-isolated COBOL programs — batch jobs without copybook dependencies, programs with documented specifications — GPT-6.1 Sol's accuracy on this evaluation is now high enough that AI-assist pipelines can be seriously evaluated rather than dismissed.

For enterprises with deeply interdependent mainframe systems — the more common case in financial services and government — the bottleneck is organizational and architectural, not model accuracy. Extracting and documenting the business logic in existing programs, cataloging copybook dependencies, and building the validation infrastructure to confirm behavioral equivalence at the system level are all prerequisites that no foundation model accuracy benchmark resolves on its own.

The COBOL workforce math makes that work urgent regardless of AI capability. With 10% of experienced COBOL developers retiring annually and the language no longer taught in US computer science curricula — the ACT-IAC report on legacy modernization noted in December 2024 that formal COBOL education has effectively ended in American universities — the window for AI-assisted migration while institutional knowledge still exists is narrowing. GPT-6's benchmark results do not change that timeline, but they do change the quality of the AI-assist layer available to organizations that choose to invest in the migration infrastructure now.

For OpenAI, the announcement signals strategic intent: that the next significant revenue opportunity in enterprise AI coding is not in modern-stack developer tooling — already served by GitHub Copilot and a crowded field of AI coding assistants — but in the harder and higher-value problem of making the legacy infrastructure of global finance and government legible and operable for the engineers who must maintain it through the next decade.

Frequently Asked Questions

Does 97.9% accuracy mean GPT-6 can migrate our COBOL system automatically?

No — and understanding why requires understanding what that 97.9% actually measures. The behavioral fidelity evaluation tests isolated COBOL programs for byte-for-byte output parity in C++, Java, Kotlin, Python, or Rust. Real enterprise COBOL systems involve copybooks (shared data definitions linking dozens of programs), JCL job streams that govern program execution, CICS transaction environments, and decades of undocumented business logic. None of those elements are present in the evaluation. GPT-6's accuracy on this benchmark means that the translation step, for well-isolated programs with clear specifications, has crossed a meaningful capability threshold. It does not mean end-to-end enterprise migration can run unattended.

How does MultiLangBench compare to existing AI coding benchmarks?

MultiLangBench is specifically designed to measure cross-language code generation on a single, comparable task across all 20 languages: build a SQLite-compatible database engine from scratch. The key design insight is that it uses language-agnostic evaluation — all implementations are checked by comparing SQL output against a reference SQLite result — rather than requiring language-specific test harnesses.

It differs from SWE-bench (which measures software engineering on real GitHub repositories) and from HumanEval (which measures single-function code generation). It's the first public benchmark specifically designed to measure low-resource programming language capability at scale. OpenAI has released the full evaluation suite on GitHub for independent verification and community use.

Is COBOL still actually used in banking?

Yes — more than most people outside enterprise IT realize. IBM estimates that COBOL powers most banking transactions globally, handling 95% of ATM transactions and 80% of in-person credit card transactions, with more than 40% of online banking systems running on COBOL. The IRS, the US Treasury, and multiple state government agencies run critical systems on COBOL.

The language's persistence reflects its reliability in high-volume transaction processing and the extreme cost and risk of replacing working systems — not neglect. The challenge is not that COBOL systems are failing; it's that the workforce with the skills to maintain them is retiring, and that the cost of maintaining them will increase as institutional knowledge disappears.

How does GPT-6 compare to IBM Watsonx Code Assistant for COBOL modernization?

They are not directly comparable — they address different parts of the migration problem. IBM Watsonx Code Assistant for Z version 2.8 is a purpose-built, mainframe-integrated agentic tool that connects to copybook libraries, job control metadata, and VSAM data definitions. It operates on the full enterprise context that makes real migration possible.

GPT-6 via the OpenAI API operates on code files without mainframe integration. GPT-6's benchmark accuracy on the behavioral fidelity evaluation — 97.9% at max reasoning effort — is a meaningful translation-step result. IBM Watsonx's structural advantage is the enterprise context access. For most large enterprise migrations, these tools will be complementary: GPT-6 for translation accuracy; IBM and similar tools for the system integration, copybook management, and validation infrastructure around the translation step.