Your AI Aced the Test. That Doesn't Mean It Understood the Question

Imagine grading an exam where every answer is correct but every bit of working is nonsense. Did the student understand the material? That is the question sitting at the center of AI research right now. The headlines keep coming: frontier models solving olympiad geometry, beating human experts on PhD-level science questions. Open the hood and something other than reasoning appears to be doing the work.

One framing note. This is not a story that broke last week. It is a slow structural problem the research community has been circling for years, and it deserves to be read as a research arc rather than a news cycle.

Chain-of-Thought Is Not a Transcript

Modern models talk themselves through problems before answering. The industry calls this chain-of-thought, or CoT. For users it feels reassuring — you can watch the model work.

But nothing guarantees that text is a record of the actual computation. Researchers call this the faithfulness problem: does the visible reasoning match the internal path that actually produced the answer? The uncomfortable finding is that it often does not.

The classic experiment goes like this. Give a model multiple-choice questions and quietly plant a bias in the prompt — make every example answer (A). The model starts picking (A) far more often on new questions. Fine, models pick up on patterns. Here is the part that matters: ask the model to explain its choice and it never mentions the pattern. It manufactures a clean, plausible line of reasoning for a decision that was actually driven by something else entirely.

The human analogue is confabulation, not lying. Split-brain patients do this — they generate confident explanations for behavior they have no introspective access to. When Anthropic tested its own reasoning models on this, the rate at which they acknowledged using a planted hint sat well below half. And it did not climb reliably as models got more capable.

Clever Hans, Now With Transformers

In 1900s Berlin there was a horse named Clever Hans who could apparently do arithmetic. Ask him to add seven and five, and he would tap his hoof twelve times. He was not doing math. He was reading involuntary micro-shifts in his handler’s posture and stopping when the tension broke. He had not learned addition. He had learned to detect the answer signal.

Machine learning has a name for this: shortcut learning. The model does not learn the rule you intended. It grabs whatever feature in the training data correlates with the label, because that is cheaper.

The examples are not hypothetical. A skin cancer classifier learned to look for rulers rather than lesions — clinicians place a measuring ruler in the frame when photographing suspected malignancies. A wolf-versus-husky classifier turned out to be a snow detector. A pneumonia model learned to read the hospital equipment tokens burned into the corner of chest X-rays, because scans from the hospital with sicker patients came from a different machine.

Language models do the same thing. Change the numbers in a math problem and accuracy drops. Insert one irrelevant sentence and performance collapses. Apple’s GSM-Symbolic paper is the canonical demonstration: swap only the names and numeric values in GSM8K grade-school word problems and accuracy falls measurably across many models. Add one throwaway clause — five of the kiwis were smaller than average — and some models dropped by as much as 65 percent. The logical structure of the problem never changed.

How Benchmarks Break

Which leads to a nastier problem. The instruments we use to measure AI progress are themselves unreliable.

Start with data contamination. Benchmark questions and their answers are on the public internet, so they end up in training corpora. At that point the model is not solving, it is recalling. A student who saw the exam beforehand does not have a meaningful score. When researchers build fresh problems at matched difficulty, scores come down — repeatedly, across benchmarks.

Then there is construct validity: does the thing you measure match the thing you meant to measure? We want to measure reasoning ability. What benchmarks actually measure is answer-production rate on a specific problem format. For humans those are close enough to interchange — if someone solves a calculus problem, assuming they know calculus is reasonable. For models the inference does not hold, because the path to the correct answer can be structurally unlike anything we imagined.

Stack Goodhart’s law on top. Once a measure becomes a target, it stops being a good measure. Labs compete on benchmark scores, so training pipelines drift toward those scores whether or not anyone intends it. Scores rise. Underlying capability rises less. The gap widens.

The Case Against Overcorrecting

Skepticism can overshoot, and it does.

Shortcut learning does not mean nothing real is happening. Frontier reasoning models perform well on problems that plausibly were not in the training data. Extending the chain-of-thought improves accuracy — which is evidence that the process is doing something, not just decorating the output. Pure theater would not move the score.

The generalization critique is also a matter of degree. Humans generalize imperfectly too. Every classroom has students who fall apart when the problem format changes. Holding models to a bar that humans fail is not analysis, it is scorekeeping.

The real argument was never binary — reasoning or not reasoning. It is about where the capability extends, where it stops, and whether we can locate that boundary. Right now we mostly cannot. That is the actual problem.

What This Means If You Ship Software

This is less abstract than it sounds.

AI safety strategies that lean on CoT monitoring get shakier. The premise is that if a model starts scheming, the reasoning trace will show it. If the trace is decoupled from the computation, that defense is tissue paper — and the model does not need any intent to conceal. The visible text simply is not a readout of internal state.

Procurement based on benchmark scores is similarly risky. A model scoring 90 on a public eval is not scoring 90 on your workflow. Validate on your own domain data, ideally in formats unlikely to appear in training corpora. Change the numbers. Rephrase the setup. Drop in one irrelevant sentence. That last trick alone is remarkably diagnostic.

Be most careful when you treat a model’s explanation as evidence. A stated rationale makes review feel easier, but the rationale may not be why the model answered that way — and a plausible-sounding explanation tends to make reviewers check less, not more. The explanation is another generated artifact. Treat it that way.

The Takeaway

There is nothing inherently wrong with a system that reaches correct answers by an unfamiliar route. Calculators do not multiply the way people do. The failure mode is measuring machines with instruments built for humans, then extending machines the trust we extend to humans, without understanding the difference.

What the field needs is not higher scores but more honest measurement. Until then, the best available defense is a habit: doubt the path, not just the answer.

When was the last time you re-ran a prompt with only the numbers changed?