Before You Replace Sonnet, Look at What You're Paying It to Do
Key takeaways
- Rules or smaller models can handle some decisions with fixed choices.
- A short answer can still require complex reasoning.
- Savings depend on the cost per resolved request, including mistakes and retries.
- A hybrid system needs to recognize which cases require a more capable model or a person.
An AI support agent may need a capable language model to write a useful reply. Does it need the same model just to decide whether to check a shipment or start a refund? Suppose you want to move that selection work from Claude Sonnet 4.6 to Amazon Strands Decider: the first question is which decisions you can sensibly hand over.
Separate the jobs before swapping the model
“Replacing Sonnet” can describe very different changes. Replacing the model that understands a customer’s situation and writes a response is a much bigger undertaking than replacing the step that selects the next tool.
Take a familiar support request: “My order hasn’t arrived.”
The system decides to check the shipment, retrieves the order information, and writes an explanation based on what it finds. There are at least two distinct jobs here: choosing an action and composing a response.
Using one model for both is convenient. It gives developers fewer components to wire together. But convenience doesn’t mean every step needs the same capabilities.
The useful unit of analysis is the individual task. A cheaper way to choose a shipping lookup tool tells you little about whether that same approach can explain a delivery problem well.
A fixed menu gives you room to simplify
Imagine the routing step has four options:
- Track a shipment.
- Start a refund request.
- Get account help.
- Contact a human agent.
It doesn’t need to write an essay. It needs to pick the appropriate action.
If the input conditions are clear, rules may be enough. If customers express the same intent in many different ways, a classifier may fit. A smaller general-purpose language model is another candidate worth evaluating.
None of these options guarantees savings. Asking a large model to return one label still requires it to read the input and make a judgment. Introducing a separate classifier can also mean maintaining training data and another operational component.
A shorter output is a formatting choice. A cheaper system requires a method that performs the task well enough at a lower total cost.
Three calls becoming one doesn’t mean a bill cut by two-thirds
Consider a hypothetical service handling 1 million requests a month, with three general-purpose model calls per request. That’s 3 million calls.
If two calls only select the next action, moving those decisions elsewhere could reduce general-purpose model calls to 1 million.
The arithmetic is appealing. The invoice is less cooperative.
Those two decisions still have to run somewhere, and their replacements have costs. The original calls may also differ in input and output length, so removing two of three calls doesn’t necessarily remove two-thirds of model spending.
Then there are mistakes. Route a routine delivery question into a refund workflow, and the system may need another check, a retry, or a human to untangle it. Each adds cost.
The better metric is cost per successfully resolved request. A lower model bill is less impressive if more customers need multiple attempts or staff intervention.
The same logic applies to speed. Measure the time to resolution alongside the time to the first response. A fast wrong turn still makes the journey longer.
A one-word answer can hide a difficult decision
“My delivery is late. If it doesn’t arrive by tomorrow, I’d like to cancel.”
This request contains both a shipping question and a conditional cancellation request. Picking a route requires understanding how those intentions relate. Treat it as a straightforward label assignment, and the system may miss the condition entirely.
Permissions and payment approvals illustrate the same problem. The output may be just “allow” or “deny,” while the decision depends on substantial context. Output length and reasoning difficulty are separate things.
A hybrid approach is worth considering. Rules or a smaller model handle clear requests. Ambiguous requests, mixed intentions, and missing information trigger escalation to a more capable model or a person.
That approach depends on recognizing difficult cases. A selector that confidently makes the wrong choice won’t necessarily escalate. Having a better model available downstream helps only if the request reaches it.
Evaluate the routing system on real inquiries. Look at the errors it makes, how reliably it flags difficult cases, and how often a request needs additional handling.
Before changing models, map what each call actually does. Separating response writing from routing can reveal where a simpler approach is sufficient and where judgment remains difficult. The useful question is whether each decision justifies the capability you’re paying for.
Comments
Loading comments...