Benchmark the Model–Harness Pair, Not the Model
A coding model's runtime can change cost by orders of magnitude without producing a dramatic pass-rate gap.
“Which model is best for coding?” sounds like a precise engineering question.
Usually, it is missing half of the system.
A coding agent is a model inside a harness that chooses tools, assembles context, serializes results, decides when to stop, and determines whether an answer counts as complete. Change that harness and the same model can become dramatically more expensive without becoming dramatically more accurate.
The deployable unit is not the model. It is the model–harness pair under a stated budget.
That makes many model-only comparisons useful for screening and weak for procurement.
A leaderboard hides the runtime that produced the score
The harness is easy to treat as plumbing. It is not.
It decides which repository files enter the context, how tool calls are represented, when old history is compacted, what the model sees after a command fails, whether completion is verified, and when the loop gives up. Each choice can alter token use, latency, failure behavior, and human oversight.
A current preprint, The Scaffold Effect in Coding Agents, isolates that variable. The researchers ran Qwen 3.6 Plus and MiniMax M2.5 through Goose, OpenCode, and OpenHands-SDK on a stratified 50-task subset of Terminal-Bench Pro.
That produced 300 trials: three harnesses, two models, and 50 tasks. The task set covered eight categories, and the authors released configurations, trial logs, aggregates, and analysis scripts.
This is preliminary work under review, not a peer-reviewed universal ranking. Its value is narrower and more practical: it holds the models fixed while measuring what the surrounding runtime changes.
Accuracy moved a little; cost moved by orders of magnitude
Within a fixed model, pass-rate differences across harnesses stayed within 0–8 percentage points. At 50 tasks per cell, most paired 95% bootstrap confidence intervals included zero. The study therefore does not establish a decisive correctness winner among the three harnesses.
Token efficiency told a different story.
Goose used 28,142 tokens per solved task with Qwen and 36,950 with MiniMax. OpenCode used 1,147,740 and 1,546,977 respectively. That is roughly a 40× difference in tokens per solved task across both models.
The gap was not explained by OpenCode taking 40 times as many turns. Average turn counts were only about 1.2 times higher. The authors attribute the larger spread to per-turn token volume, context growth, tool serialization, and harness-specific accounting, while noting that the available accounting is not perfectly symmetric.
The practical point survives that limitation. A pass-rate table can make two systems look close while one consumes orders of magnitude more context to reach each accepted result.
Failure modes belonged to the scaffold, not just the model
The same failure fingerprints repeated across both models.
Goose failures were dominated by reasoning stops. OpenHands-SDK more often reached verification failures or maximum-turn exits. OpenCode showed idle-loop and timeout behavior. OpenCode also averaged roughly ten times more no-action turns than Goose.
Replication across two models supports the authors' interpretation that these patterns came from the harness. It does not prove every version of those tools will behave that way on every workload. But it shows why “the model failed” is often an incomplete diagnosis.
If a run repeatedly rereads files, serializes oversized tool results, or continues after the useful work has stopped, buying a stronger model may not fix the waste. The problem can live in context policy, stopping rules, or verification logic.
That changes incident review. Teams should classify trace-level exits such as no-action turns, verifier rejection, maximum-turn exhaustion, tool errors, and unsupported completion. A single pass/fail label erases the mechanism required to improve the system.
The strongest model-first argument is still reasonable
Model leaderboards are not useless. When one harness is fixed, they are exactly the right tool for comparing models under a shared scaffold. A stronger model can improve reasoning, tool selection, and repair quality without any runtime change.
Model-only reporting is also simple. “Model X scored Y” is legible to buyers and reproducible when the missing configuration is stable and documented.
The problem begins when products use different scaffolds but the result is still attributed to the model name. The comparison now mixes model capability with context management, tool interfaces, budgets, and stopping behavior.
The scaffold-effect paper even found model upgrades and harness changes produced pass-rate shifts of comparable scale in its sample: 4–10 points for the model and 0–8 for the harness. On cost, however, the harness shifted tokens per solved task by about 40 times, while the model change moved it by roughly 1.0–1.3 times.
So the model-first view is strongest as an early filter. It becomes incomplete when the question changes from “which model appears capable?” to “which deployed agent should this team run?”
Evaluate the pair in two passes
The cleanest internal evaluation freezes one half of the system at a time.
First, choose a model that clears the minimum capability bar. Then freeze the model, environment, task set, and acceptance tests while varying the harness. Measure the complete operating profile:
| Measure | Why it belongs in the decision |
|---|---|
| Solved-task rate | Whether the system completes the workload |
| Tokens per solved task | Cost of accepted outcomes, not attempts |
| Wall-clock time | Developer wait and queue capacity |
| No-action turns | Idle-loop waste hidden by pass rate |
| Verifier and max-turn exits | Harness-specific failure behavior |
| Human corrections | Oversight burden after the run |
Twenty representative internal tasks can be more decision-relevant than hundreds of unrelated public tasks, provided the environment and acceptance checks are fixed before the comparison. Run paired trials because agent behavior is stochastic. Inspect traces because aggregates cannot tell you whether a failure came from reasoning, tools, context, or termination.
Only then freeze the chosen harness and compare model candidates. This does not eliminate confounding, but it prevents a wasteful runtime from being credited to the model it happens to contain.
The verdict
A model leaderboard can tell you which models deserve an evaluation. It cannot tell you what a deployed coding agent will cost, how long it will stall, or which failure mode a reviewer will inherit.
The honest engineering claim is longer: this model, inside this harness, under this token and latency budget, solved this workload with these failure modes.
That sentence is less marketable than “best coding model.” It is also the comparison a production team can actually use.
