# The Upgrade Isn't Always Up: Why We Rolled a Whole Agent Fleet Back a Version > Source: > Published: 2026-08-04 00:00:00+00:00 # The Upgrade Isn't Always Up: Why We Rolled a Whole Agent Fleet Back a Version August 4, 2026 by Asif Waliuddin The Upgrade Isn't Always Up Editor's note: the external numbers in this piece are real and cited — the harness studies, the regression reports, the vendor's counter-claims. I've left in the evidence that contradictsme, on purpose. A story that only quotes the data agreeing with it isn't a story, it's a brochure. The unglamorous decision Last week I did something that felt slightly heretical. A newer, more capable version of the model my agents run on had been available for a while. On paper it's better — better reasoning, better scores on the hardest agentic-coding benchmarks, adaptive thinking, same price. Every instinct in tech says *upgrade*. I rolled my fleet back to the older version instead. Not the product teams — they were never moved. Just my chief-of-staff agents, the ones that run governance, hold the portfolio's context across a long day, and coordinate a dozen other agents. On those, the newer model was, in my lived experience, *worse* — not at any one task, but at the thing that's hardest to benchmark and easiest to feel: staying coherent inside its own scaffolding over a long, messy, real session. The older version just… held the thread better. So I reverted. Config-only, kept the newer model one flag away as a rollback route, and moved on. Then I got curious about whether I'd fooled myself. What the data says — including the part that disagrees with me I went looking for evidence on both sides, because "the new one felt worse" is exactly the kind of thing a person talks themselves into. Here's what's actually out there. **The part that backs me up is bigger than I expected.** There's now a real body of work showing that the *harness* — the scaffolding around the model, how context is managed, how tools are described, how errors are caught — often matters more than which model you pick: - Cursor's benchmarking found the **same model scoring 46% on one harness and 80% on another.** - A Stanford / Tsinghua study measured performance gaps of **up to 6×** from harness design alone. - On SWE-bench, the same model with different scaffolding swings **10–20 percentage points.** And "the newer version regressed" is not a feeling I invented — it's a documented, repeating pattern elsewhere. The clearest analog is one team's report that a newer model was **"stronger at planning, weaker at execution and instruction adherence"** — better analysis, worse operational result. That is almost word-for-word what I felt. Other reports: a coding agent 4–7× slower on tasks the older sibling did smoothly; a version that misbehaved so badly a team rolled it back to an earlier one to get their work done. **Now the part that disagrees with me.** The vendor's own launch materials and several review sites say the newer model is the *more* consistent one — less run-to-run variance, and specifically *better* at holding context and recovering from its own mistakes over long, 20-plus-step agent sessions. That is the exact opposite of what I experienced. I'm not going to bury that. If I'm right, I have to explain why the benchmark says otherwise. The reconciliation: it's fit, not tier Here's the resolution, and it's the reason this is worth writing down. Both things can be true at once, because **"which model is better" is the wrong question. The right question is which model fits the harness you already built.** Three reasons the aggregate benchmark and my N-of-1 can both be honest: - **My harness was tuned around the older model.** Its effort defaults, its verification habits, the little quirks my scaffolding learned to expect — all of that was shaped over months of the older model's behavior. Drop a differently-behaving model into a wrapper built for a different one and it can underperform its*own*benchmark potential. That's not the model being bad. That's a fit problem. - **Benchmarks measure a different job than mine.** Coding-agent benchmarks measure coding. My agents mostly do governance, coherence, and portfolio self-awareness across a long horizon. A model can genuinely win the coding benchmark and still lose*my*specific workload. - **The evidence isn't weighted equally.** The harness-matters findings come from independent research (Cursor, Stanford, SWE-bench). Several of the "the new one is more consistent" claims come from the vendor and from SEO-flavored review sites. I trust the first kind more than the second, and I try to say so out loud rather than pick whichever supports me. The lesson I'm keeping The tech-industry reflex is that newer is better and you upgrade or you fall behind. Sometimes that's true. But for agentic systems specifically, the model is one component in a machine you built, and a machine has *fit*, not just parts. A moderately-capable model in a harness tuned for it will beat a more-capable model the harness doesn't fit — and the research now backs that up with 6× and 46-versus-80 numbers, not vibes. So the discipline I'm taking away isn't "distrust new models." It's: **Treat every model change as a migration, not a free upgrade.** Gate it. Keep the rollback route.**Measure it on your actual workload,** not the leaderboard for a job you don't do.**When your lived experience fights the benchmark, don't just defer to the benchmark — and don't just defer to your gut either.**Find out*why*they disagree. The answer is usually more useful than either number. I reverted a whole fleet against the vendor's own benchmark current. I might re-run the comparison in a month with the harness re-tuned and find the newer model wins after all — and if it does, I'll say so just as plainly. That's the whole point. The goal was never to be right about a version number. It was to keep the machine coherent, and to stay honest about how I know. *— Asif Waliuddin, NextGen AI*