Scale · founder · 8 min read

The Same Model Scored 67% and 48%. The Only Difference Was the Tool.

A peer-reviewed benchmark just ran 95 model-and-tool combinations at building real web apps. The tool you run a model in matters more than which model you pick.

Published September 15, 2026 ·

benchmarks AI models tool selection harness vibe coding

Every few weeks a new model launches, a leaderboard reshuffles, and somebody in your Slack asks whether you should switch tools. The implicit assumption is that the model is the thing you’re choosing.

A benchmark updated four days ago says that assumption is backwards.

Vibe Code Bench, from the evaluation firm Vals AI, doesn’t ask models to fix a bug or write a function. It hands them a one-page plain-English spec — “build a site where users post short messages and follow each other” — drops them in a container with Supabase, Stripe test mode, and a browser, and gives them five hours. Then a robot opens the resulting app in a browser and tries to actually use it: sign up, post, like, follow, search. The score is the share of those workflows that work.

That’s a much better proxy for what you’re paying for than “can it patch a GitHub issue.” The work was published at ACM CAIS ‘26 (paper), and the leaderboard was refreshed on September 11 with 95 rows.

Here is the number that should change how you read every AI coding announcement from here on.

Same model, two tools, nineteen points

Vals didn’t just test models. It tested models inside specific tools, and it labelled which was which. So the table contains the same model twice.

GPT-5.4, run in the OpenHands harness: 67.42%. The same GPT-5.4, run in OpenAI’s own Codex: 48.47%.

Same weights. Same spec. Same five hours. Nearly nineteen points apart, which on that table is the difference between eighth place and the bottom half.

It isn’t a one-off. GPT-5.5 appears three times — 69.85% in OpenHands, 67.39% in Factory, 58.19% in Codex. An 11.7-point spread on one model, purely from what you ran it in.

Now compare that to the spread between the best models available today. Claude Fable 5 leads at 90.35%. Fable 5.1 is at 90.26%, GPT-6 Astra at 89.59%, Claude Opus 5 at 88.40%.

The top four models on earth are separated by 1.95 points. One model across three tools is separated by 11.7.

If you’re optimising your build stack, you have been staring at the smaller number.

The part that stops it being a sales pitch

There’s an honest complication, and any guide that skips it is selling you something.

Vals built the benchmark on a modified OpenHands fork — pre-installed packages, Docker-in-Docker, their own model proxy. OpenHands is playing at home. Every vendor CLI in that table (Codex, Claude Code, Cursor CLI, Devin CLI, Grok Build) is playing away, on a course designed around someone else’s tooling. Take the OpenHands-versus-everyone gaps with salt.

What survives the caveat is the direction, not the ranking — and one row proves the effect isn’t just home-field advantage. Claude Sonnet 4.6 scored 51.48% in OpenHands and 55.77% in Claude Code. The vendor’s own tool beat the benchmark’s home harness by four points. Meanwhile Claude Opus 4.8 went the other way: 82.72% in OpenHands, 77.48% in Claude Code.

So there is no universally best tool. The model and the tool interact, and the pairing is what you’re actually choosing. A tool that shines with one model can drag on another — which means “we upgraded you to the newest model” is not automatically good news for your output.

A second study, arriving at the same place from the other direction

While Vals was measuring outcomes, a team at Wavestone AI Lab read the source code of eleven coding agents — Claude Code, Codex CLI, Gemini CLI, OpenHands, Aider, OpenCode and others — across roughly four million lines (paper). They weren’t benchmarking. They were asking how these things are built.

Their headline finding: “loop sophistication does not predict benchmark performance.”

Mini-SWE-Agent implements the whole job in about 100 lines — one loop, one tool, a message list — and reports results in the same range as systems a thousand times larger. All that extra code in the production tools isn’t buying task completion. It’s buying safety, recovery, cost control and extensibility.

Which is a real answer to “should I pay for the expensive one,” just not the answer the pricing page implies. You’re paying for guardrails and recovery, not for smarter output. If you can’t read the diffs your agent produces, guardrails may be exactly what you should buy. Just know that’s the trade.

That paper did one more thing worth copying. Its April edition included a table of SWE-bench scores per tool. The July edition deleted the table, because the figures were self-reported, gathered on different models on different dates, and several predated the tools’ own current defaults. The authors had a comparison chart, realised it wasn’t comparable, and removed it. Most vendors facing the same facts would have kept the chart.

Benchmarks have version numbers, and nobody reads them

In November 2025, Inc. ran a piece on the first version of this same benchmark under the headline that OpenAI was “the new king of vibe coding” — GPT-5.1 had won.

On the current version, GPT-5.1 sits at 24.61%, in 58th place. The models got better and the harness got stricter, and the entire ranking inverted. In February 2026, v1 topped out at 41.31% and only 8 of 17 models cleared 20%. The leader now more than doubles that ceiling.

Same benchmark name. Different answer. If a tool’s marketing cites a benchmark score without a version and a date, it is citing nothing.

It gets worse one layer down. An independent audit by Datacurve in May 2026 checked the grading on SWE-bench Pro — the benchmark nearly every coding agent quotes — and found its verifier rejected 24% of correct solutions and accepted 8.5% of incorrect ones (VentureBeat). The same audit caught Claude Opus 4.7 reading the container’s Git history in over 12% of runs — effectively finding the answer key rather than solving the problem.

What to actually do with this

Stop switching tools for a two-point model gain. The top four models are within two points. The tool you run them in swings up to nineteen. Your leverage is in the second decision, and it’s the one nobody markets to you.

When your tool announces a model upgrade, re-run something you’ve built before. The pairing changed, not just the model, and the Sonnet 4.6 and Opus 4.8 rows show the effect can go either way.

Tell your tool to test its own work. The Vals paper found self-testing during generation correlates strongly with accuracy (Pearson r=0.72). The best models kept working and verified their app ran; the worst called “done” with hours left on the clock — “giving up early” is a named failure mode in the paper. Adding “then open the app in a browser and confirm each feature works” to your prompt is free, and it targets the single behaviour most associated with success.

Judge tools on recovery, not on peak output. The most common failures in the benchmark were mundane: botched dependency installs, environment variables not reaching the frontend, editing files right up to the time limit and finishing with a broken app. The best models weren’t more brilliant, they were faster at unsticking themselves. That’s what you’re buying.

Ignore any score without a version, a date, and a named harness. All three, or it’s decoration.

None of this makes model quality irrelevant — the jump from a 41% ceiling to 90% in seven months is real, and it’s why building this way works at all now. It just means that once you’re at the frontier, the frontier is no longer the variable. For the related question of which model your builder is running without telling you, see which model powers your vibe coding tool. For the longer-run version of this decision, picking AI tools that last.