Your AI Subscription Needs More Than a Launch-Day Score

Key takeaways

• Tracking quality requires repeated tests under consistent conditions.
• Lower scores alone cannot establish that a provider deliberately reduced performance.
• Results by task can reveal changes that an overall average hides.
• An AI subscription’s value depends on the time it saves after checking and rework.

If Claude breezed through a routine task yesterday but keeps asking for clarification today, the monthly bill starts to feel more conspicuous. That frustration deserves attention, but your memory of its best-ever answer makes a lousy benchmark.

Start by defining the failure

Saying Claude has gotten worse is a claim about results. Saying it has been “nerfed” adds a claim about intent.

The gaming term usually means someone deliberately weakened something. Repeated failures on a familiar task might justify investigating a decline. They cannot, by themselves, tell you what the provider changed or why.

Even “worse” needs unpacking. Shorter answers, more factual errors, ignored formatting instructions, and slower responses are different problems. Lumping them together makes it harder to identify what changed.

Track accuracy, instruction following, and response time separately. “Has it started copying numbers into the wrong table cells?” gives you something concrete to investigate. “Why is it so dumb now?” mostly gives you somewhere to put your frustration.

The useful benchmark continues after launch

A launch score captures performance under particular test conditions. It cannot guarantee that next month’s subscription will handle your work equally well.

What subscribers need is longitudinal evaluation: repeating comparable measurements over time.

Imagine a small set of recurring tasks. Extract fields from a document. Fix a bug in a piece of code. Summarize a passage while following specific constraints. Record whether each task succeeded, which requirements were missed, and how long the response took.

The conditions matter as much as the questions:

• Keep prompts and attached materials consistent.
• Match the conversation setup, including whether each test starts in a fresh chat.
• Record whether search or other tools were used.

You also need to decide what you are evaluating. A test of the underlying model’s capabilities has a different scope from a test of the subscription app.

For someone paying to get work done in that app, a failed tool call belongs in the quality assessment. It still leaves the task unfinished.

Repetition helps, but averages can hide the problem

Generative AI can produce different answers to the same question. One failed attempt does not establish a decline.

An evaluation could run 10 work tasks three times each per day. That is an illustrative test design, not a reported experiment. Repeating each task helps reveal ordinary variation, which you need to understand before interpreting differences between dates.

Even then, the overall score can mislead.

Suppose summaries improve while numerical extraction gets worse. The combined average might barely move. Someone who spends all day extracting numbers would still have a legitimate complaint.

Results should therefore be broken down by task, with examples of the actual failures. A document-processing test also tells you little about capabilities it never measured, such as debugging or analyzing long arguments.

Repeated measurements can strengthen the evidence that performance changed. Explaining the cause requires more: the evaluation conditions and a record of relevant service changes. A downward line on a chart does not explain itself.

Count the work left on your desk

Subscribers expect to finish similar work with a reasonably consistent amount of effort. Model names and usage limits help describe a product, but they do not tell you whether it will meet that expectation.

Suppose a draft arrives just as quickly as before, but checking its facts and fixing its prose now takes twice as long. Your effective cost has increased, even if the subscription price has stayed put.

A benchmark could miss that change. Your calendar will not.

Useful quality reporting should show success rates for different tasks, variation across repeated attempts, and meaningful changes to the service. Subscribers should be able to see when something changed and which capabilities were affected.

That does not require promising a correct answer to every question. It requires explaining what level of quality users can reasonably expect under stated conditions—and telling them when that expectation needs to change.

Whether Claude has actually gotten worse is a question for comparable records over time. Whether it is still worth paying for also depends on the work it leaves behind. The subscription earns its keep through the hours you get back.