Anthropic Put Claude Through a Riemann Zeta Gauntlet. The Results Say More About AI Evals Than About Math

It’s been over a year since AI models started collecting math olympiad gold medals. Mathematicians remain conspicuously unimpressed. Their objection is simple: solving problems and doing mathematics are not the same activity. Anthropic just waded directly into that gap, testing Claude’s mathematical ability in one of the nastiest corners of the field — the Riemann zeta function.

One caveat up front. This hasn’t generated much community discussion yet. Searching the last 30 days of Reddit turns up little. So this isn’t a roundup of what the internet thinks. It’s an argument about why the underlying question matters.

Why the Zeta Function Is the Right Weapon

The Riemann zeta function sits at the center of the Riemann Hypothesis, the most famous unsolved problem in mathematics. It governs the distribution of prime numbers. What makes it interesting here is that it’s a nearly perfect test bed for evaluating AI.

Three reasons. First, it’s genuinely hard. Undergraduate calculus gets you nowhere. You need to actually understand complex analysis, analytic continuation, functional equations. Second, the answers are unambiguous. Specific values and properties of the zeta function are long established, so there’s no debating whether a model got it right. Third — and this is the one that matters — it’s an ideal environment for confident nonsense. The literature around the Riemann Hypothesis is enormous: papers, blog posts, lecture notes, forum threads. A model can recombine those sentences into something that reads beautifully and fools anyone who isn’t a specialist.

That third condition is the whole point. Benchmark scores can’t distinguish understanding from pattern matching. The zeta function is a good knife for separating them.

Doing Math vs. Being Good at Computation

There’s a critique mathematicians keep repeating about these models: they are excellent at reproducing known paths and weak at inventing new ones.

Mathematical research roughly runs through defining the problem, hunting for counterexamples, designing lemmas, constructing the proof, and verifying it. LLMs are strongest at the formal parts of those last two stages — applying theorems they already know, grinding through computation, assembling an argument into readable prose. What they’re still bad at is the move that says “this problem gets easy if you reframe it as a different problem.” Reformulation. That’s the hard part.

Which means evaluation has to change too. Accuracy alone is the wrong metric. You need to look at why the model chose a given approach, how it routes around a dead end, whether it notices when it’s wrong. Watching a model work on zeta-function problems surfaces exactly this. Sometimes it lands the right answer and the reasoning still smells memorized. Sometimes it gets the answer wrong while the approach is mathematically healthy.

A Broken Proof Doesn’t Look Broken

Hallucination in mathematics is worse than in almost any other domain. Code crashes when you run it. Factual claims get caught by a search. A proof, though, arrives looking flawless with one wrong line buried in the middle. Catching that line requires someone who already knows the mathematics at that level.

The neighborhood of the Riemann Hypothesis is especially mined. The internet is littered with claimed proofs, and the overwhelming majority of them are wrong. When that material seeps into training data, the model learns something specific: how to state a false result with total confidence.

Hence the clear direction of travel — pairing models with proof checkers. Force output into a theorem prover like Lean and a machine verifies the proof. Fluency stops mattering the moment it fails to compile. It’s the most practical route around the weaknesses of natural-language proof.

What It Means When a Frontier Lab Grades Its Own Model

Worth stepping back here. This is Anthropic evaluating Anthropic’s model.

That’s not automatically bad. Only the people who built a model can see inside it. What it trained on, where it collapses, what its failure patterns look like — the lab knows best. Anthropic has invested in interpretability research for years and has a track record of publishing papers that concede model limitations openly.

The structural problem persists anyway: the people choosing what to test are the people who built the thing being tested. Once you decide which problems go on the exam, you’ve already determined a large share of the outcome. You might unconsciously pick domains where your model shines. You might file the weak domains under “early-stage research.” None of this requires bad faith.

Which is why work like this gets more valuable when it’s paired with outside verification. Independent mathematicians posing the same problems differently. Other labs’ models sitting the same exam. Someone checking whether the results reproduce. A lab’s internal research is a good starting point, not a conclusion.

So, Does AI Do Math?

The honest answer right now is: partially.

Computation and formal manipulation are already at competent-graduate-student level. Applying known theorems, carrying a long argument without slipping — that works. What doesn’t yet work is defining a problem fresh, feeling that something is off, using intuition to rule out a direction that would waste twenty years. That last category happens to be the most valuable part of mathematical research.

But this split doesn’t look stable. As mathematicians start using AI as a computational assistant and a counterexample hunter, the boundary keeps moving — the line between the creative work humans keep and the labor-intensive work AI absorbs. Some researchers are already using LLMs as assistants on real papers.

The Takeaway

The zeta function makes an excellent litmus test. The answers are fixed, but the road to them separates understanding from memorization. That Anthropic looked at this territory at all suggests frontier labs are starting to care about something beyond benchmark scores.

One thing still nags, though. When the organization that built a model evaluates that model’s capabilities, how much weight should the results carry? Measuring AI’s mathematical ability turns, at that point, into a question about the credibility of the entire AI evaluation ecosystem. We may need systems for verifying answers more urgently than we need AI that produces them.