Opus 5 is the new cool kid on the block, beating Fable 5 in benchmarks, yet remaining strangely frustrating to work with in practice. In order to gain more insight into Opus 5's behavior, and to see how it performed on my repo, I ran Opus 4.8 and Opus 5 on the same 25 tasks drawn from merged work in Stet's own repository. These ran one per model per task with medium reasoning and identical evaluation criteria.
TL;DR
- The score tied: 9/25 strict test passes each: the same 8 tasks, plus one unique pass apiece.
- Opus 5 searched wider and verified more. It used more shell commands on 18 of 25 tasks, more test commands on 15, and performed more revision passes on the files it touched.
- Opus 4.8 stayed contained. It had a smaller patch footprint on 20 of 25 tasks, meaning it stayed closer to the change that was actually merged.
- Costs landed in the same range: Opus 5 was ~1.4% cheaper on the typical task, with ~4% more tokens and ~4% longer wall-clock.
High level the results look the same, both models passing 9 tasks. But, within these passes, neither the patches nor the process to get there looked the same.
Opus 4.8 had a lower task footprint (measure of how much code changed compared to the merged change) on 20 of 25 tasks. Opus 5 ran more shell commands on 18, more test commands on 15, and touched more files on 12 while tying on 11. Total tool calls split almost evenly, 13 to 11 with one exact tie. The models spent nearly identical interaction budgets on opposite parts of the work: Opus 4.8 spends its budget on the edit; Opus 5 spends its budget discovering what to edit, and how to validate that change.
This difference is why it's important to look beyond top-level pass rates. A test pass rate simply tells you whether the test suite accepted the final patch. It notably does not tell you how the agent searched, what it chose to verify, how much code it left for review, whether it ever reached the file that owned the requested behavior, or how maintainable the code it wrote is.
A test fail can also hide a materially correct patch that still behaves as intended. So, Stet runs a second check called equivalence, asking whether the agent patch made the same behavioral change as the merged patch did, even when the underlying implementation differs? Equivalence moves both models the same way. Opus 4.8 was judged equivalent on 12 of 25 tasks and Opus 5 on 11, with both equivalent on 10: the 8 shared test passes plus 2 shared test failures (stet-66762b4c
, stet-f0caded9
) where both patches did the merged change's work but missed something that made tests pass. Under either lens, the models stay effectively tied.
*Note: this is 25 matched tasks from one repository. What follows is a behavioral read of a few tasks, not a definitive model ranking. *
Grading #
The deterministic testing signal and the grader signals point in different directions. Footprint risk separates the two models cleanly: 20 of 25 pairs for Opus 4.8. Double cliking on this, the extra code is not coming from just volume of additional tests: 40% of Opus 5's churn lands in test and fixture code versus 34% for Opus 4.8, and its non-test churn is still 1.46× larger. When our graders do pick up signal, they lean towards Opus 5 on the coherence, instruction adherence, edge-case handling, and maintainability dimensions.
Looking at this data, we can put together coherent hypothesis about what wider search and heavier test execution buy in practice. Judged patch quality that tilts slightly upward while the artifact surface tilts sharply upward. Still worth reminding that at this sample size both leans are directional.
Every task, side by side #
Aggregates hide individual anecdates that are usefule in understanding model behavior. Let's dive deep on a few!
Three things stand out. The split here is broad, and is not just driven by a few outliers: Opus 4.8's footprint advantage spans both small fixes and large multi file changes. On the five widest tasks, the footprint gap alone exceeds Opus 4.8's average footprint across the whole cohort (0.24). And, among the five tasks where Opus 5 left the smaller footprint includes the most consequential one: stet-2450ca2d
. The only task Opus 5 passed while Opus 4.8 failed.
Opus 4.8 stayed closer to the patch it first understood #
Footprint risk is Stet's deterministic measure of patch surface: files touched, churn, size, and overlap with the merged diff. A lower footprint score means that the agent's patch is more similar to wat was merged previously. It says nothing about correctness, only surface.
stet-89dfbc27
shows why containment can be valuable. The task was to restore ignored files to Stet's synthetic base commit. Both agents found the production fix: add --force
to git add -A
.
Opus 4.8 changed one production file, added no test, and passed. Opus 5 made the same production change and then added a 141-line end-to-end test. Its test compiled and exercised a real boundary. It also turned a small repair into a much larger surface. Opus 5 spent nearly three times as long and 83% more recorded cost to produce the same accepted implementation plus broader verification.
stet-2450ca2d
required two new test-file patterns in internal/gitops/testclassifier.go
. Opus 4.8 edited internal/validate/footprint_risk.go
, an adjacent consumer of the classifier output. It tested the function it changed, but never reached the owner of the requested behavior. Opus 5 found testclassifier.go
, added both patterns, and passed strict and equivalence evaluation. Equivalence asks whether the agent patch made the same material behavioral change as the merged patch, rather than only whether deterministic testing passed.
Opus 4.8's patch was centered around the wrong owner. Note what else this task shows: it is one of only five where Opus 5 left the smaller footprint. When Opus 5's broader search finds the right owner, its wider exploration does not necessarily translate into a bigger patch.
A broader version of the same failure appears in stet-5d766dde
. Opus 4.8 implemented the network-posture portion but omitted the paired schema/cache bump, mount stripping, and standalone resolver.
In summary, Opus 4.8's trajectory profile pays off when the task boundary is already known. It becomes more risky when the hard part is discovering how many owners the task actually has, and where that surface is, which is exactly the situation many large enterprise codebases find themselves in.
Opus 5 searched wider and kept working after the first edit #
Looking per-task, Opus 5 recorded more shell calls on 18 pairs, more test commands on 15, and more distinct patched files on 12 with 11 ties. Patch operations per file divides patch calls by distinct patched files on each task, so a higher means more revisions per touched file, not better work.
Total tool calls are almost perfectly balanced between the two models. Opus 5 did not consume more interactions. It allocated more of them to the shell, test execution, and repeated editing.
That broader route is what passed stet-2450ca2d
: six test commands instead of three, and the search continued past the adjacent consumer to the owning classifier. The implementation was small once the correct owner was found. The meat of the task was repository navigation to find the right surface.
The wider route created different failure modes on larger changes.
In stet-bbbbae09
, Opus 5 recorded 24 patch calls across 8 files, renamed one required test, and omitted another. Opus 4.8 made 15 patch calls across 6 files and cleared the strict evaluator.
A longer trajectory is not waste, and a shorter one is not efficiency. Opus 5 often finished sooner and cheaper, yet missed named acceptance artifacts after more revision. Opus 4.8 passed the evaluator, but its review artifact still raised API and authority concerns. Neither patch generalizes beyond its task.
stet-6f84e978
shows the valuable side of expansion. Opus 5 ran seven test commands against Opus 4.8's two and added a preservation test for an explicit non-Rust obligation. The stronger verification took 34.9 minutes instead of 6.1, while recorded cost rose only from $1.11 to $1.18. Wall time, tokens, cache mix, and price measure different parts of the trajectory.
Opus 5's wider search sometimes found the missing owner and sometimes created more room to drift from an exact contract. You can only see this when the comparison keeps the trajectory and the patch, not just the final test result.
Time, tokens, and cost split in different directions #
Three resource measurements answer three different questions. Agent duration is wall-clock time from the run's start to finish. Total tokens combine recorded input and output, including cached input. Cache-aware cost applies each model's price schedule to fresh input, cached input, and output. Opus 4.8 finished sooner on 17 pairs, Opus 5 cost less on 15, and the typical-task cost estimate landed just below Opus 4.8 at −1.4%.
Opus 5 used fewer tokens on 16 of 25 pairs and cost less on 15, so the counts lean its way. The paired-geometric magnitude points the other way on tokens: on the pairs where Opus 5 used more, it used enough more to put its typical task token estimate 4.3% above Opus 4.8, while cost settled 1.4% below and duration ran 3.7% longer. The count says how often a direction occurred; the paired estimate says how large the typical change was with every task weighted equally.
Two shared passes show how wide the range is:
- On
stet-15439c21
, Opus 5 finished a small deletion in 294 seconds, 488K tokens, and $0.42 — 3.3 times faster with 2.4 times fewer tokens than Opus 4.8. Both passed. - On
stet-89dfbc27
, Opus 5 added a large end-to-end test and used 70% more tokens, 83% more cost, and 2.8 times the duration. Both passed.
The tails lean one way. On four of 25 tasks, Opus 5 used more than 2.5 times Opus 4.8's tokens, peaking at 4.1 times on stet-e928166f
. Opus 4.8's largest token excess in the other direction was 2.4 times. On stet-66762b4c
, Opus 5 produced substantially more implementation and test code and ran 10 minutes longer, yet used fewer recorded tokens and dollars; both remained strict failures, and both cleared a separate adapted-reference check that tested the material behavior without revising the strict result.
There is no clean "faster model" or "cheaper model" in this cohort. Resource use follows what the agent decides to inspect, implement, and verify on each task.
What the eval doesn't see #
The thing that seriously frustrates me (and everyone else I talk to) about Opus 5 day to day is its extremely verbose and hard to parse prose, which doesn't appear in these numbers at all. This evaluation scores the artifact: the patch, the tests it ran, the trajectory of how the agent got there. It does not score the interaction modes with the agent that produced that result. Walls of explanation, the restated plans, the summaries of summaries, eyes glazing over, LGTM, ship it. None of the eight craft dimensions measures how much reading the human had to do to get the final patch.
Code side verbosity, another noted issue with Opus, does actually show up in our footprint risk metric. Even so, Opus can be disciplined in its patches and still exhausting for interaction, and this evaluation is structurally blind to that. This is an artifact eval, not a collaboration eval.
The more agentic model #
On these tasks, Opus 5 looks like the more agentic model. It performed broader searching of the repo to figure out the correct surface before committing to an edit, it went looking for the place that owned the behavior instead of patching the nearest consumer, and it decided to validated its own work, resulting in more test commands and more post-edit revision, rather than stopping at the first patch that seems right. And, it did all of that while staying in the same price range: cheaper on 15 of 25 tasks, about 1.4% cheaper on the typical one.
The cost of that behavior shows up in review surface rather than dollars: 20 of 25 tasks left a bigger patch that a human (supposedly) has to review. Opus 5 buys discovery and verification, and you pay in patch surface and a little wall-clock.
Despite the prikly personality, I'll be using Opus 5, or having Fable delegate to it, for my hardest and most demanding problems.
Again, this is n=1 repository. Model choice is one harness lever alongside instruction files, skills, tools, and reasoning settings, and any of them can change how an agent searches, edits, tests, and stops. The decision belongs on your own merged work, where the task distribution represents your own challenges, and the code review costs are tangible.
Methodology #
Every task is derived from work that was actually merged into Stet's own repository — a PR or commit, replayed from a frozen snapshot of the tree as it stood before that change, with the issue prompt and the evaluation commands carried along. Both models ran all 25 tasks in the same Claude Code harness, one attempt per model-task cell at medium reasoning, against identical evaluation criteria.
The pass/fail score counts a cell as a pass only when the selected tests accept the agent patch. The adaptive lower bound additionally counts a cell whose implementation diverged from the merged change but was judged to reach the same behavior. Equivalence asks whether the agent patch made the same material behavioral change as the merged human patch, rather than only whether commands passed. Footprint risk is deterministic, not judged: it scores patch surface from files touched, churn, size, and overlap with the merged change, and lower means less code for a reviewer to hold in their head. It says nothing about correctness. The eight craft dimensions and the code-review rubric are pointwise judge scores from claude-sonnet-4-6, paired per task under a 0.25-point draw band on the 0–4 scale.
FAQ #
How did Opus 4.8 and Opus 5 score on these 25 matched tasks?
Each model recorded 9/25 strict functional passes and a separately reported 10/25 adaptive functional lower bound. Eight tasks passed for both, 15 failed for both, and each model had one unique strict pass.
How did the models behave differently?
Opus 4.8 had lower footprint risk on 20 of 25 pairs. Opus 5 recorded more shell calls on 18 pairs, more test commands on 15, and more patched files on 12 with 11 ties. These are local paired observations, not global model traits.
Was Opus 5 cheaper or faster?
Opus 5 cost less on 15 of 25 pairs and used fewer tokens on 16, but Opus 4.8 finished sooner on 17. The paired-geometric estimates put Opus 5 1.4% below on cost, 4.3% above on tokens, and 3.7% longer on duration; the canonical report carries no decision-grade uncertainty interval for these.
Which model should a team use?
This comparison supports a local routing hypothesis, not a universal default: test Opus 4.8 on contained work where the owner is known and patch surface is expensive, and test Opus 5 on discovery-heavy work with exact contract verification. Validate that split on your own merged changes.