AI Ported COBOL to Java. It Ported the 30-Year-Old Bugs Too

There are still billions of lines of COBOL running the world’s banks, insurers, and government benefit systems. The people who wrote it have mostly retired. So the obvious idea arrives on schedule: just point an LLM at it. And the models do the job — sometimes impressively well. They also carry over every bug in the original, with total fidelity.

The Translation Succeeded. That’s the Problem

Let’s be clear about what this is not. It isn’t a story about AI failing to convert code. It’s the opposite.

The traditional bar for a legacy migration is behavioral equivalence. Same inputs, same outputs. If the 30-year-old COBOL program and the shiny new Java service agree on every test case, you shipped it.

But what if the original is wrong? Then reproducing the wrongness is exactly what equivalence demands. And LLMs hit that bar cleanly. Rounding logic that drops a fraction of a cent in the wrong direction. An overflow that only triggers past a certain threshold. A branch that quietly returns zero when it should raise an error. All of it ports. All of it passes. It’s wrong in precisely the same way the source was wrong.

Translating Is Not Understanding

An LLM translates code. It does not read the code, work out what the system was supposed to do, and rebuild it. That gap is the entire story.

Watch what a human engineer does when they hit something strange in a COBOL module. They stop. They go find someone and ask why it’s written that way. The answer is almost always one of three things: it’s a bug, it’s a regulation nobody remembers, or it’s a carve-out built for one large customer in 2003.

The model never asks. Its job is to render the code in front of it into the target language, and it does. COBOL also happens to be one of the worst possible inputs for inferring intent — comments are sparse, and the ones that exist are two-line notes from someone who left the company during the Clinton administration. No context, no basis for judgment.

Here’s the part that should worry you more. The Java that comes out looks great. Modern idioms, sensible class boundaries, readable names. So it sails through code review. In the COBOL original, that weird 40-line paragraph gave off a smell — reviewers slowed down when they hit it. Translation launders the smell away. The bug survives; the warning sign doesn’t.

Nobody Left Who Can Tell a Bug From a Spec

This is the real trap: does anyone in the organization still know whether a given odd behavior is a defect or a requirement?

Financial systems are full of these. A calculation truncates at a specific decimal place, and it turns out that was a regulatory requirement written in the 1980s that nobody has looked at since. Or the reverse — something is unambiguously broken, but three downstream systems were built assuming the broken output, so fixing it breaks the quarter-end close.

Telling these apart takes domain knowledge, not code reading. The model only has the code. And here’s the uncomfortable part: the organization commissioning the migration has often lost that knowledge too. That’s why the COBOL was never retired in the first place.

Which produces a specific, absurd outcome. COBOL that nobody understands becomes Java that nobody understands. The language changed. The problem did not.

What Teams Are Actually Trying

Groups that have hit this wall are converging on a few approaches.

Flag the anomalies instead of smoothing them over. Don’t ask the model only to translate — ask it to produce a separate list of logic that looks suspicious, unreachable, or inconsistent with the surrounding code. LLMs are considerably better at pointing at weird code than at deciding what to do with it. Turning that into a review queue is a much better use of the capability than trusting the output wholesale.

Run both systems in parallel against production data. Shadow the COBOL with the Java for a few months on real traffic and diff the outputs. Every divergence is a review item. The catch is obvious: this only surfaces cases where the two disagree. The faithfully translated bug produces identical output in both systems, so it passes this test as well. Shadow running is necessary. It is not sufficient.

Rewrite from a reconstructed spec. Recover what the system is supposed to do as a document, then build against that document instead of against the old code. This is the only approach that actually removes the inherited bugs. It’s also the most expensive, and spec reconstruction is precisely the thing the organization already couldn’t do — which is how you end up back where you started.

In practice most teams land on some blend of the first two. Let the model do the translation, then treat the output as a draft for review rather than a deliverable. That framing alone changes how much scrutiny the Java gets.

This Was Never Really About COBOL

Step back and the language stops mattering.

Python 2 to Python 3. AngularJS to React. On-prem to cloud. Same shape every time. The model is good at moving what exists. It has no opinion on whether what exists is correct.

The bottleneck in migration used to be the porting work. That part is genuinely getting solved. The bottleneck has moved to deciding what shouldn’t be ported at all — and that decision doesn’t come from reading code. It comes from people who remember why.

The Takeaway

Almost every benchmark for AI code migration measures one thing: how closely the output matches the behavior of the original. If the original is wrong, a higher score is a worse result. That’s an uncomfortable property for a metric to have.

So think about the legacy system in your own org. Point at one strange behavior in it. Is there still someone who can tell you why it’s there? If the answer is no, the tooling isn’t the first thing that needs fixing.