OpenAI's COBOL Migration Numbers: 5.4% to 86.6%. What the Evals Measure, and What They Do Not

OpenAI says a recent model went from 5.4% to 86.6% accuracy on its COBOL migration evaluation, at lower cost. That is a big jump. It is also OpenAI's own eval, run on OpenAI's own models.

This article reads OpenAI's October 19 post, Advancing Coding in Low-Resource Programming Languages. All numbers are OpenAI's. I have not run either evaluation.

What OpenAI says it built

The post says COBOL, Fortran and Visual Basic have less code and documentation for models to learn from than Python or JavaScript, yet support critical systems. It cites COBOL as supporting more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions, and Fortran as used in weather forecasting and nuclear record safety analysis. Those are OpenAI's cited figures, linked to outside sources in the post, which I did not check.

It describes two evaluations:

• MultiLangBench: can a model build a SQLite-compatible database engine from scratch in one of 20 languages?
• COBOL migration: can a model migrate existing COBOL programs to C++, Java, Kotlin, Python or Rust while preserving behavior?

MultiLangBench results

How it works, per the post: the candidate program receives an initial database file and a SQL query and produces a modified database. OpenAI uses the reference SQLite implementation to check the contents, which lets the same tests apply to any language. Tests are adapted from SQLite's existing suite. It reports test pass rate and API cost.

At maximum reasoning effort:

• GPT-6 Astra: 87.4% average pass rate across 20 languages, up from 26.8% for GPT-5.6 Sol. OpenAI says every language improved.
• Per language, Sol to Astra: COBOL 10.9% to 72.1%, OCaml 25.6% to 85.8%, Fortran 21.4% to 83.5%, Visual Basic 27.5% to 89.4%. Astra scores above 83% in 19 of the 20 languages, and COBOL stays the hardest at 72%.
• GPT-6 Luna, the lower-cost option: 40.3% average, with an estimated $0.77 per sample versus $3.48 for Sol, about 78% lower.

OpenAI says it is releasing the evaluation setup, test suite and per-language results on GitHub. I did not look at the repository, so I cannot say how the released version compares with the post.

COBOL migration results

The task: migrate programs to one of five target languages with hidden tests that need byte-for-byte parity on observable outputs. A migration counts as successful only if all tests pass.

The post's example of why this is hard: COBOL's fixed-point arithmetic discards excess decimal digits unless rounding is requested. A rewrite that rounds can run without errors and still give different financial results.

Results: GPT-6.1 Sol reaches 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting, at 61% of the API cost (the intro of the post says 39% less, the same number). At higher reasoning effort, accuracy reaches 97.9%, which OpenAI says is close to saturating the evaluation.

What the post itself cautions

OpenAI states that applying this to production software "still requires expert review and testing against the system's business rules and dependencies." It also says its evaluations help it see where models "still fall short." The post does not describe what the migration programs look like, how many there are, or how they were chosen, in the text I read.

What I would check before trusting it

These are my own untested suggestions.

1. Test set size and shape. A near-saturated eval with a small set of programs can hide failures on real systems with decades of patches.

2. Parity is only for what tests observe. Byte-for-byte parity on hidden tests is strong, but it covers what the tests cover. Rounding, record handling and file formats outside the tests are not covered.

3. Cost framing. The costs are per sample on OpenAI's tasks. A real migration involves reading, review and test building around the model.

4. Your own sample. Pick a few of your own programs, with known tricky arithmetic, and run them through before planning a project.

5. Reasoning effort. The 86.6% is at low effort and 97.9% at higher effort. The post does not give the cost of the higher setting in the text I read.

What I would not conclude

• That COBOL migration is solved. OpenAI says the eval is close to saturation, not that production migration is.
• That the lower-cost model is good enough. Luna scores 40.3% on MultiLangBench, well below Astra.
• That results from one lab's benchmarks compare across vendors. The post compares only OpenAI's own models.

Frequently asked questions

What is MultiLangBench?

An OpenAI evaluation where models build a SQLite-compatible database engine from scratch in 20 languages, checked against the reference SQLite.

What was the headline COBOL result?

GPT-6.1 Sol at 86.6% accuracy on the COBOL migration eval at low reasoning effort, versus 5.4% for GPT-5.6 Sol.

How does a migration count as successful?

All hidden tests must pass, with byte-for-byte parity on observable outputs.

Which target languages does it test?

C++, Java, Kotlin, Python and Rust.

Can I run MultiLangBench myself?

OpenAI says it is releasing the setup and test suite on GitHub. I did not check the repository.

Does this mean I can migrate my COBOL system with AI now?

OpenAI says production use still needs expert review and testing against business rules.

Why does COBOL arithmetic matter?

Per the post, fixed-point arithmetic truncates unless rounding is requested, so a rewrite that rounds can silently change financial results.

Sources

• OpenAI, Advancing Coding in Low-Resource Programming Languages (October 19, 2026): all claims and numbers above.