How Much Work Does SWE-2 Leave for You?

Key takeaways

• Benchmark scores need context, including tool access, time limits, and allowed attempts.
• Useful coding AI must make changes that fit an existing codebase and preserve its behavior.
• Total cost includes model fees, human review, and rework.
• A coding specialist’s value depends on how well it handles your team’s recurring tasks.

A coding agent can deliver a patch quickly and still leave you with an afternoon of cleanup. When comparing Cognition SWE-2, Fable 5.1, and GPT-Astra, the useful question is how much work remains before that patch is ready to accept.

Read the conditions next to the score

A coding benchmark measures performance on a defined set of problems. Those might involve writing a function or fixing a bug in an existing repository.

The evaluation conditions matter as much as the headline score. Did the model get one attempt? Could it run the code, inspect failures, and try again? Which tools could it use, and how much time did it have?

Think of a live coding interview versus a take-home assignment. Both involve programming. You would still want to know which one produced the result before comparing candidates.

Apply the same scrutiny to SWE-2, Fable 5.1, and GPT-Astra. Solving the same problems is only part of a fair comparison. The time budget, available tools, and number of attempts also need to line up.

A leaderboard can help build a shortlist. Turning its scores into a prediction about your team’s productivity takes more evidence.

Give it a codebase with history

Writing a working feature is only part of the job. That feature also has to coexist with everything already running.

Suppose you ask an agent to add discounts to an online store. It calculates the discounted price correctly, but canceled orders now receive the wrong refund. The happy path works. The task is unfinished.

A useful evaluation should check whether the model follows the repository’s conventions, preserves existing behavior, and handles relevant edge cases. Its explanation should also make the changes easy for a reviewer to understand.

Pay particular attention to change scope. A small bug fix that arrives with a broad restructuring creates more code to inspect and more behavior to check. The reviewer now has to work out which changes were necessary.

A large diff is easy to produce. Earning confidence in it takes time.

Measure the time until the task is done

Fast responses feel productive. To find out whether they actually save time, keep the clock running through review and corrections.

Consider two hypothetical results, assuming the work happens sequentially:

These are illustrative numbers, not measurements of SWE-2 or its competitors. They show how the apparent winner changes when you measure the whole task.

The same logic applies to cost. A lower model fee can be outweighed by repeated attempts and lengthy review. A more expensive model may justify its price if it leaves substantially less work for the developer.

Record review and rework time alongside response time and model charges. That gives you a clearer view of what each completed task costs.

Test the work your team keeps doing

“Specialized for coding” is a claim worth testing against specific tasks. Bug fixes, feature development, and test writing place different demands on a model.

Choose recent examples from your team’s work. Give each model the same starting code, requirements, tools, and time limit.

Then evaluate the results in a consistent order:

• Does the requested behavior work?
• Do existing features still behave correctly?
• How much did a person have to fix?
• Do the tests meaningfully check the requested behavior?

That last question deserves attention. Passing tests only helps if those tests cover what matters.

The results may also point to different choices for different teams. A team that spends much of its week fixing small bugs should care about how reliably a model finishes those fixes with little human intervention. Strength elsewhere may have less practical value.

SWE-2 earns its place in a workflow by reducing the effort needed to complete and review the work that team actually does. Keep the benchmark score in view, but keep the clock running through the last correction. That is when you find out how much time you got back.