Your New Model Is Smarter and Somehow More Annoying

Every major model launch runs the same script. The benchmark numbers clear the previous generation by a comfortable margin. Then the people who’ve actually been using it for a week show up to say the old one was better. Opus 5 got the same treatment. What makes this worth writing about isn’t the model — it’s that this has happened with GPT-4 Turbo, Claude 3.5, and every significant Gemini update over the past three years. Same complaint, different weights.

One caveat up front: I went looking for a specific, quantifiable thread to anchor this and didn’t find one worth citing. So rather than narrate a live argument, this is about the mechanism underneath it — why the argument keeps coming back regardless of which lab ships what.

Benchmarks Measure the Wrong Half of the Job

Almost every benchmark is a single-shot test. SWE-bench, the closest thing coding has to a standard, hands the model one GitHub issue and asks for a patch. Clear problem, one correct answer, unambiguous scoring. That’s what makes it a benchmark.

Actual work with a model looks nothing like that. You give it a vague requirement. You get back something 60 percent right. You say “no, not like that” three times. What matters in that loop isn’t first-try accuracy — it’s steerability. How fast does it catch your intent? When you push back, does it fix the thing you pointed at, or does it rewrite three functions you liked?

Benchmarks barely touch that axis. Which creates a real gap where scores climb and the experience degrades. A more sophisticated first attempt is worse than a crude one if it’s sophisticated in the wrong direction and harder to unwind.

Diligence Has a Cost

Recent models across every lab share one trait: they think more, verify more, and explain more. On safety and accuracy, that’s a straightforward win.

It’s also cumulative. Ask for a small refactor and you get five files read first, an impact analysis, and a request to confirm before proceeding. Once, that reads as careful. Forty times in a workday, it reads as friction.

There’s a line that shows up constantly in developer threads: the old model was wrong, but it was fast. Getting a wrong answer in five seconds and fixing it yourself genuinely beats a correct answer in ninety — especially if you already know the codebase and the model is essentially typing for you.

You’re Comparing Against a Version That Doesn’t Exist

The variable most people leave out of these arguments is their own baseline.

Six months with a model means six months of internalizing its failure modes. You know which prompts land, which requests to route around, when to break a task in half. That tacit knowledge doesn’t transfer. A new model resets it. Equal raw capability still feels worse, because you’ve lost the interface you built in your head.

And the comparison itself is rigged. The thing you’re benchmarking the new model against isn’t the old model at launch — it’s the old model after you spent half a year training yourself on it. Nobody remembers being frustrated in week one. That’s not a fair fight, but perception doesn’t care.

Some of It Is Real

So far this is mostly a story about cognitive bias. That doesn’t mean every reported regression is imaginary.

Infrastructure changes. Launch traffic spikes, inference resources get rebalanced, context handling gets adjusted, system prompts get quietly revised. Providers rarely announce any of it. Which means “this worked yesterday and doesn’t today” is sometimes literally true.

The problem is that it’s unprovable at the individual level. Sampling means the same prompt produces different output every time, so the person reporting degradation can’t produce a clean reproduction, and the provider answers with eval scores. Both sides are correct by their own evidence and neither is engaging with the other’s. That’s why these threads burn a hundred comments and settle nothing.

Build Your Own Eval, Seriously

There’s one way out of this, and it’s unglamorous: keep a personal eval set.

It doesn’t need to be rigorous. Pick ten tasks you actually do in your actual codebase — the refactor you run weekly, the test-writing pattern, the gnarly debugging case — and run them against each new model. This beats any public leaderboard for your purposes, because it’s weighted by your work distribution instead of someone else’s.

Give it a week minimum. The first two days of friction are almost entirely adaptation cost, not capability loss. Whatever’s still bothering you on day seven is the actual signal.

When a new model feels worse, it might be worse, or your habits might be overfit to the last one. Only data separates those. Most of us are running on vibes and calling it evaluation — and vibes have no memory, so we relitigate this from scratch every launch. Write it down instead. The next release will be a lot easier to judge.