Before You Trust an AI Coding Score, Check the Denominator

Key takeaways

• Real-SWE reports a top solve rate of 38.8% across 10 private enterprise tasks.
• Ten tasks scored once each as pass or fail cannot produce a 38.8% result.
• Private code can reduce concerns about exposure during training, but its origin alone does not establish workplace relevance.
• Adoption decisions also need to account for human review time and the cost of recovering from failures.

An impressive coding demo leaves one question unanswered: can the agent finish a useful job in your company’s codebase? Specific Labs’ Real-SWE tackles that question with private enterprise tasks and a reported top solve rate of 38.8%. Before that number becomes a slide in someone’s procurement deck, it deserves a closer look.

Start with the denominator

If you run 10 tasks once each and mark every result as a success or failure, the possible scores move in 10-percentage-point steps. You can get 30% or 40%. You cannot get 38.8%.

That does not make the reported score wrong. It means the calculation involves something beyond that simple setup.

Repeated runs, weighted tasks, or partial credit could produce a different percentage. These are possible explanations, not established details of Real-SWE’s methodology.

Reading the result as “roughly four out of ten tasks solved” therefore risks changing its meaning. The number of attempts and the aggregation method matter.

So does the word top. Which agents and configurations were compared? Did they receive the same number of attempts and the same execution budget? A highest score becomes useful for comparison only when you understand the conditions behind it.

Private code helps answer one question

Public coding benchmarks raise a familiar concern: the model may have encountered the problem, its solution, or both during training. A strong result can mix problem-solving ability with familiarity.

Private enterprise code can reduce that concern. But “private” alone does not prove that the evaluation material never overlapped with training data. How the code and tasks were handled still matters.

Company repositories also offer opportunities to test skills that isolated coding exercises may miss. Consider a hypothetical payment bug. Fixing it could require tracing existing exception handling, understanding the test setup, and finding assumptions scattered across several files.

That is valuable work. Writing the patch may be the easy part.

Whether Real-SWE tests those abilities depends on its actual task selection and constraints. The provenance of a repository tells you where the work came from; the evaluation design tells you what the agent had to do.

“Solved” has to survive code review

In a software team, a successful fix usually carries several obligations. The bug must stop occurring. Existing behavior must keep working. The change must be understandable enough for someone else to review and maintain.

A passing test suite can be part of that evidence. It is not automatically the whole case.

Imagine a patch that passes tests by silently swallowing an error. That could still be unacceptable in production. Conversely, an agent that spots an ambiguous requirement and asks a useful clarification question may be doing sensible engineering, even if a benchmark grading only final code gives it little credit.

These examples make the success criteria central to interpreting a solve rate. Does success mean passing tests? Does a human assess whether the requirements were met? Are regressions checked?

Tasks drawn from real companies can support a convincing evaluation. Their value depends on how closely the assessment preserves the demands of the original work.

Ten tasks can reveal failures, but cannot represent every team

A small evaluation can still be informative. Ten complex tasks, examined carefully, may reveal exactly where an agent loses context or makes a damaging assumption.

Repeating those tasks can show how consistently it succeeds. It does not expand the range of companies, codebases, or engineering problems being tested. Broader claims require broader coverage.

For an adopting team, another measurement matters: human effort saved.

If reviewing and repairing an agent’s patch takes longer than making the fix directly, the practical gain shrinks. An incomplete attempt could still help if it accurately narrows down the cause of a bug. A binary success score can miss both outcomes.

Real-SWE’s 38.8% needs a clear denominator, explicit success criteria, and a closer look at the tasks before it can serve as a workplace scorecard. Teams should put review time and failure recovery costs beside the solve rate. The useful question is how much work remains when the agent says it is done.