Your Harness Didn’t Get Better, It Got Lucky: Noise Bands for Agent Evals

Paolo Perrone's October 31, 2026 analysis in Data Science Collective, recomputing public Terminal-Bench 2.0 leaderboard data (89 tasks, at least 5 runs each), found that swapping the harness from Terminus 2 to Copilot CLI under Claude Opus 4.6 moved scores by -2.5 points (62.9% to 60.4%), well inside a noise band of plus or minus 6.7 points, while swapping the model from Claude Opus 4.6 to Kimi k2.5 produced a 20.4-point drop that exceeded the plus or minus 7.8-point band. Perrone's recomputation found only 39 of the 89 tasks passed on every run, 30 were flaky and 20 failed every run, and that shrinking the harness-swap band to plus or minus 2.5 points would require roughly 650 tasks at 5 runs each.

Picture a pull request that changes the harness around your coding agent: a new system prompt, a different tool layout, maybe a whole new CLI wrapped around the same model. The description says “eval score up 3 points.” Somebody approves it. Two weeks later somebody else reverts half of it because “the score dropped 2 points.” Nobody in that loop ever asked the only question that matters: is a 3-point move bigger than what this eval does on its own when nothing changes? I have been on both sides of that review, and I no longer accept a harness claim without an answer to it. This article is about how to get that answer from numbers other people have already published, and how I would wire the same calculation into a .NET tool and a CI gate. I am not reporting any experiment of my own here. Every number below comes from a named source, and I say who produced it. The cleanest worked example I have seen this autumn is in Paolo Perrone’s “How to Build an Agent Eval You Can Trust” Data Science Collective, October 31, 2026 . He took the public Terminal-Bench 2.0 leaderboard data, which publishes at least 5 runs for every one of the suite’s 89 tasks, and recomputed two comparisons himself. He is explicit that these are his own figures: he writes “by our count.” So read everything in the table below as an author-run recomputation of public leaderboard data. I have not re-derived it. +----------------------------------+---------------+---------------+---------+------------+---------------+| Comparison | Before | After | Change | Noise band | Reading |+----------------------------------+---------------+---------------+---------+------------+---------------+| Harness swap, Claude Opus 4.6: | Terminus 2 | Copilot CLI | -2.5 pt | +/-6.7 pt | inside band: || Terminus 2 - Copilot CLI | 62.9% | 60.4% | | | can't call it |+----------------------------------+---------------+---------------+---------+------------+---------------+| Model swap under Terminus 2: | Opus 4.6 | Kimi k2.5 | -20.4pt | +/-7.8 pt | outside band: || Opus 4.6 - Kimi k2.5 | 62.9% | drop only | | | real drop |+----------------------------------+---------------+---------------+---------+------------+---------------++--------------------------------------------------------------+-------------------------+| Terminus 2 + Claude Opus 4.6, 89 tasks, 5+ runs each | Value |+--------------------------------------------------------------+-------------------------+| Solid tasks passed every run | 39 || Flaky tasks passed some runs | 30 || Broken tasks failed every run | 20 || Share of runs passed | 62.9% || Share of tasks passed on every run | 43.8% || Band floor even with unlimited runs harness swap | about +/-5.7 pt || Tasks needed at 5 runs each to shrink band to +/-2.5 | about 650 || Best-to-worst spread, Factory Droid + Opus 4.6, 5 full runs | 3.4 pt |+--------------------------------------------------------------+-------------------------+ Source: Paolo Perrone, “How to Build an Agent Eval You Can Trust,” Data Science Collective, Oct 31, 2026, recomputed by the author from the harborframework/terminal-bench-2-leaderboard data on Hugging Face “by our count” . The Kimi k2.5 absolute score is not given in the article, only the drop. Author-run figures, not independently verified. Two things in there changed how I read leaderboards. The first is the contrast itself. A 2.5-point move from swapping the harness sits comfortably inside a band of plus or minus 6.7. A 20.4-point move from swapping the model is nearly three bands away. One of those is a finding. The other is a coin that landed slightly differently. The second is the 3.4-point spread for a single configuration run five times. The same harness, the same model, the same tasks, and the full-suite score still moved 3.4 points between its best and worst run. That is already more than the harness swap. If one configuration can wobble that much against itself, a single run of each side of a comparison tells you almost nothing. If you have used BenchmarkDotNet, you already have the right instinct. It never prints just a mean. It prints the mean next to an error column, and nobody on a .NET team would merge “my change makes this method 2% faster” when the error column says 5%. Agent evals deserve exactly the same discipline, and the arithmetic is not much harder. Here is the calculation Perrone uses, in plain steps. 1. Rerun every task. Five runs per task is the Terminal-Bench 2.0 leaderboard’s minimum. For each task you get a pass rate: 0 out of 5, 1 out of 5, and so on up to 5 out of 5. With five runs, a task’s pass rate can only be 0, 20, 40, 60, 80 or 100 percent. That coarseness is worth remembering. 2. Pair by task. For every task, subtract its pass rate under the old configuration from its pass rate under the new one. Perrone’s example is circuit-fibsqrt: 5 of 5 under Terminus 2, 0 of 5 under Copilot CLI, so its difference is minus 100 points. You now have 89 numbers, one per task. 3. The average of those differences is the score change. For the harness swap, that is minus 2.5 points. This is the mean of per-task rates. When every task has the same number of runs, it equals the “share of all runs passed” figure on a leaderboard. 4. Measure how much the differences scatter. That is the standard deviation of the 89 differences. A few tasks swung by 100 points, most did not move at all, and the standard deviation summarizes that spread in one number. 5. Divide by the square root of the task count. This is the part people find magical, and it is just averaging. One task tells you little. The average of 89 tasks is much steadier than any single task, and it gets steadier in proportion to the square root of how many you average, not in proportion to the count. Four times the tasks halves the band. That is why Perrone’s estimate of roughly 650 tasks to get down to plus or minus 2.5 points is not a typo. 6. Multiply by 1.96. If the per-task differences behave roughly like independent draws, about 95 percent of the time the true average change sits within 1.96 standard errors of what you measured. That gives the band: band = 1.96 stddev per-task differences / sqrt number of tasks If the measured change is inside the band, the eval cannot tell the change apart from luck. That is not the same as saying the change did nothing. It says this eval, at this size, cannot see it. A quick sanity check I like to do when reading anyone’s band: run the formula backwards. A band of plus or minus 6.7 over 89 tasks implies a standard deviation of the per-task differences of roughly 32 points my arithmetic, from his published figures . In words: individual tasks are swinging by a third of the scale between the two harnesses, in both directions, and those swings almost cancel out. The headline “minus 2.5” is a small residue left over from a lot of very large, opposite movements. That is not what most people picture when they read “the score dropped a bit.” Each difference compares a task with itself. A brutally hard task and a trivial task are both measured against their own baseline, so task difficulty drops out of the band. If you compared the two configurations’ overall scores as if they were unrelated samples, task difficulty would sit inside the noise and the band would be wider. Pairing is free precision, and it only requires that both configurations ran the same tasks. My tool below refuses to compare tasks that are missing on either side for that reason. Perrone separates two sources of noise, and I think this is the most practically useful part of his piece. Run luck: the same task passes on one run and fails on the next. More runs per task shrink this. Task luck: your 89 tasks stand in for all the work you care about, and a different 89 would give a different score. More runs do nothing for this. Only more tasks help. For the harness swap, he estimates run luck at only about a quarter of the band’s variance. Even with unlimited reruns, the band would bottom out around plus or minus 5.7 points, still wider than the 2.5-point drop. I find this genuinely sobering. The instinct when a result is ambiguous is “let’s rerun it a few more times.” For this comparison, that would not have settled anything. The bottleneck was the size of the task set. Perrone’s second move is to stop treating the suite as one number and sort the tasks by what their five runs did: For Terminus 2 with Claude Opus 4.6, his count is 39 solid, 30 flaky, 20 broken. The configuration passed 62.9 percent of its runs, but only 43.8 percent of tasks passed every time. He also points out that if every task passed independently at that same 62.9 percent rate, only about 10 percent would pass all five 0.629 to the fifth power . The real suite has far more solid and far more broken tasks than that, which means the pass rate is not one coin flip repeated, it is three different populations mixed together. That matters for harness work because only the flaky tasks swing from run to run. A harness change that “improves the score” by tipping a couple of flaky tasks over on a lucky day is the classic false win. And the reverse failure exists too: under Copilot CLI, two tasks lost all five of their passes circuit-fibsqrt and schemelike-metacircular-eval, per Perrone . A verdict on the average, even a correctly banded one, hides those. I want a tool that prints them by name so somebody reads their traces before calling two harnesses equal. Perrone’s own noise floor.py is behind a paywall, and I have not seen it. What follows is my own C implementation of the formula as he describes it in the article, plus the solid, flaky and broken split. It has no dependencies beyond the base class library. It takes a CSV with one row per task, per run, per configuration: FORMAT EXAMPLE ONLY: obviously fake placeholder rows, not results from any eval.task id,run index,config,passedexample-task-a,1,baseline-harness,1example-task-a,2,baseline-harness,0example-task-a,1,candidate-harness,1example-task-a,2,candidate-harness,1example-task-b,1,baseline-harness,0example-task-b,2,baseline-harness,0example-task-b,1,candidate-harness,0example-task-b,2,candidate-harness,1 That example only shows the shape. Fed to the tool as-is, it is rejected on purpose: two runs per task is below the default minimum of five, and the tool will not compute a band from tasks that do not meet it. A few decisions I made deliberately: // NoiseBand.cs// Paired noise band for agent eval results: is a score change bigger than luck?//// Input: a CSV with one row per task per run per configuration:// task id,run index,config,passed// passed accepts 1/0, true/false, pass/fail case-insensitive . Lines starting with ' ' are ignored.// Task ids and config names must not contain commas.//// Run as a .NET 10 file-based app:// dotnet run NoiseBand.cs -- results.csv --baseline <config --candidate <config --gate improved|no-regression|none // --min-runs 5 --z 1.96 //// Exit codes: 0 = gate passed or no gate , 1 = gate failed, 2 = bad input.using System.Globalization;using System.Text;try{ var cli = Cli.Parse args ; using var reader = new StreamReader cli.CsvPath ; var runs = NoiseBand.ReadCsv reader ; var result = NoiseBand.Compare runs, cli.Baseline, cli.Candidate, cli.MinRuns, cli.Z ; Console.Write NoiseBand.Format result ; bool gatePassed = cli.Gate switch { "improved" = result.Verdict == Verdict.AboveBand, "no-regression" = result.Verdict = Verdict.BelowBand, = true, }; if cli.Gate = "none" Console.WriteLine gatePassed ? $"GATE PASSED {cli.Gate} " : $"GATE FAILED {cli.Gate} : the change does not clear the noise band in the required direction." ; return gatePassed ? 0 : 1;}catch Exception ex when ex is FormatException or ArgumentException or InvalidDataException or IOException { Console.Error.WriteLine $"error: {ex.Message}" ; return 2;}public sealed record RunResult string TaskId, int RunIndex, string Config, bool Passed ;public enum TaskClass { Solid, Flaky, Broken }public enum Verdict { InsideBand, AboveBand, BelowBand }/// <summary Pass counts for one task under one configuration.</summary public sealed record TaskStats string TaskId, int Runs, int Passes { public double PassRate = double Passes / Runs; // Solid: passed every run. Broken: failed every run. Flaky: anything in between. public TaskClass Class = Passes == Runs ? TaskClass.Solid : Passes == 0 ? TaskClass.Broken : TaskClass.Flaky;}/// <summary One task seen under both configurations. Diff is candidate minus baseline, as a fraction.</summary public sealed record TaskDiff string TaskId, TaskStats Baseline, TaskStats Candidate { public double Diff = Candidate.PassRate - Baseline.PassRate;}public sealed record ComparisonResult string BaselineConfig, string CandidateConfig, IReadOnlyList<TaskDiff Paired, IReadOnlyList<string UnpairedTasks, IReadOnlyList<string UnderRunTasks, double BaselineScore, double CandidateScore, double MeanDiff, double StdDevOfDiffs, double HalfWidth, double Z, Verdict Verdict ;public static class NoiseBand{ private static readonly string Columns = "task id", "run index", "config", "passed" ; public static IReadOnlyList<RunResult ReadCsv TextReader reader { string? headerLine; int lineNo = 0; do { headerLine = reader.ReadLine ; lineNo++; } while headerLine is not null && headerLine.Trim .Length == 0 || headerLine.TrimStart .StartsWith ' ' ; if headerLine is null throw new InvalidDataException "CSV is empty." ; var header = headerLine.Split ',' .Select h = h.Trim .ToLowerInvariant .ToArray ; var index = Columns.ToDictionary c = c, c = Array.IndexOf header, c ; var missing = index.Where kv = kv.Value < 0 .Select kv = kv.Key .ToList ; if missing.Count 0 throw new InvalidDataException $"CSV header is missing column s : {string.Join ", ", missing }." ; var rows = new List<RunResult ; var seen = new HashSet< string, int, string ; string? line; while line = reader.ReadLine is not null { lineNo++; if line.Trim .Length == 0 || line.TrimStart .StartsWith ' ' continue; var cells = line.Split ',' ; if cells.Length = header.Length throw new InvalidDataException $"Line {lineNo}: expected {header.Length} cells, got {cells.Length}." ; string task = cells index "task id" .Trim ; string config = cells index "config" .Trim ; if task.Length == 0 || config.Length == 0 throw new InvalidDataException $"Line {lineNo}: task id and config must not be empty." ; if int.TryParse cells index "run index" .Trim , NumberStyles.Integer, CultureInfo.InvariantCulture, out int run throw new InvalidDataException $"Line {lineNo}: run index '{cells index "run index" }' is not an integer." ; bool passed = ParsePassed cells index "passed" .Trim , lineNo ; // A duplicated row would silently count one run twice, so refuse it. if seen.Add task, run, config throw new InvalidDataException $"Line {lineNo}: duplicate row for task '{task}', run {run}, config '{config}'." ; rows.Add new RunResult task, run, config, passed ; } return rows; } private static bool ParsePassed string value, int lineNo = value.ToLowerInvariant switch { "1" or "true" or "pass" or "passed" = true, "0" or "false" or "fail" or "failed" = false, = throw new InvalidDataException $"Line {lineNo}: passed value '{value}' is not 1/0, true/false or pass/fail." , }; /// <summary /// Paired comparison over tasks seen in both configurations with at least minRuns runs each. /// Band half-width = z sd per-task differences / sqrt number of paired tasks . /// </summary public static ComparisonResult Compare IEnumerable<RunResult runs, string baseline, string candidate, int minRuns = 5, double z = 1.96 { if string.Equals baseline, candidate, StringComparison.Ordinal throw new ArgumentException "Baseline and candidate must be different configurations." ; if minRuns < 1 throw new ArgumentException "min-runs must be at least 1." ; var all = runs.ToList ; var a = Stats all, baseline ; var b = Stats all, candidate ; if a.Count == 0 throw new ArgumentException $"No rows for baseline config '{baseline}'." ; if b.Count == 0 throw new ArgumentException $"No rows for candidate config '{candidate}'." ; var unpaired = a.Keys.Except b.Keys .Concat b.Keys.Except a.Keys .Order StringComparer.Ordinal .ToList ; var underRun = a.Keys.Intersect b.Keys .Where t = a t .Runs < minRuns || b t .Runs < minRuns .Order StringComparer.Ordinal .ToList ; var paired = a.Keys.Intersect b.Keys .Except underRun .Order StringComparer.Ordinal .Select t = new TaskDiff t, a t , b t .ToList ; int n = paired.Count; if n < 2 throw new ArgumentException $"Need at least 2 paired tasks with {minRuns}+ runs in both configs; found {n}." ; double mean = paired.Average p = p.Diff ; double variance = paired.Sum p = p.Diff - mean p.Diff - mean / n - 1 ; double sd = Math.Sqrt variance ; double half = z sd / Math.Sqrt n ; // A difference exactly on the edge of the band counts as inside: the conservative reading. var verdict = mean half ? Verdict.AboveBand : mean < -half ? Verdict.BelowBand : Verdict.InsideBand; return new ComparisonResult baseline, candidate, paired, unpaired, underRun, paired.Average p = p.Baseline.PassRate , paired.Average p = p.Candidate.PassRate , mean, sd, half, z, verdict ; } private static Dictionary<string, TaskStats Stats List<RunResult runs, string config = runs.Where r = r.Config == config .GroupBy r = r.TaskId .ToDictionary g = g.Key, g = new TaskStats g.Key, g.Count , g.Count r = r.Passed ; public static string Format ComparisonResult r { static string Pts double fraction = fraction 100 .ToString "+0.0;-0.0;0.0", CultureInfo.InvariantCulture ; static string Pct double fraction = fraction 100 .ToString "0.0", CultureInfo.InvariantCulture + "%"; string Counts Func<TaskDiff, TaskStats pick = string.Join " / ", Enum.GetValues<TaskClass .Select c = $"{c.ToString .ToLowerInvariant } {r.Paired.Count p = pick p .Class == c }" ; var sb = new StringBuilder ; sb.AppendLine $"Paired tasks: {r.Paired.Count}" ; sb.AppendLine $"Baseline {r.BaselineConfig}: mean per-task pass rate {Pct r.BaselineScore } {Counts p = p.Baseline } " ; sb.AppendLine $"Candidate {r.CandidateConfig}: mean per-task pass rate {Pct r.CandidateScore } {Counts p = p.Candidate } " ; sb.AppendLine $"Change: {Pts r.MeanDiff } pts, noise band +/-{ r.HalfWidth 100 .ToString "0.0", CultureInfo.InvariantCulture } pts " + $" z={r.Z.ToString CultureInfo.InvariantCulture }, sd of differences { r.StdDevOfDiffs 100 .ToString "0.0", CultureInfo.InvariantCulture } pts " ; sb.AppendLine r.Verdict switch { Verdict.AboveBand = "Verdict: ABOVE the band. The improvement is unlikely to be luck alone.", Verdict.BelowBand = "Verdict: BELOW the band. The drop is unlikely to be luck alone.", = "Verdict: INSIDE the band. The eval cannot tell this change from noise.", } ; // A verdict on the mean hides single tasks that flipped completely. List them for trace reading. var flips = r.Paired.Where p = p.Baseline.Class = p.Candidate.Class && p.Baseline.Class = TaskClass.Flaky && p.Candidate.Class = TaskClass.Flaky .ToList ; foreach var f in flips sb.AppendLine $" Flipped: {f.TaskId} {f.Baseline.Class.ToString .ToLowerInvariant } - {f.Candidate.Class.ToString .ToLowerInvariant } " + $" {f.Baseline.Passes}/{f.Baseline.Runs} - {f.Candidate.Passes}/{f.Candidate.Runs} " ; if r.UnpairedTasks.Count 0 sb.AppendLine $" EXCLUDED {r.UnpairedTasks.Count} task s present in only one config: {string.Join ", ", r.UnpairedTasks }" ; if r.UnderRunTasks.Count 0 sb.AppendLine $" EXCLUDED {r.UnderRunTasks.Count} task s below the minimum run count: {string.Join ", ", r.UnderRunTasks }" ; if r.Paired.Count < 30 sb.AppendLine $" WARNING: {r.Paired.Count} paired tasks. With this few, z={r.Z.ToString CultureInfo.InvariantCulture } understates the band; consider a t value." ; return sb.ToString ; }}internal sealed record Cli string CsvPath, string Baseline, string Candidate, string Gate, int MinRuns, double Z { public static Cli Parse string args { if args.Length == 0 || args 0 .StartsWith "--" throw new ArgumentException "usage: NoiseBand <results.csv --baseline <config --candidate <config --gate improved|no-regression|none --min-runs 5 --z 1.96 " ; var opts = new Dictionary<string, string StringComparer.OrdinalIgnoreCase ; for int i = 1; i < args.Length; i++ { if args i .StartsWith "--" || i + 1 = args.Length throw new ArgumentException $"Unexpected argument '{args i }'." ; opts args i 2.. = args ++i ; } string Required string key = opts.TryGetValue key, out var v && v.Length 0 ? v : throw new ArgumentException $"--{key} is required." ; string gate = opts.GetValueOrDefault "gate", "none" .ToLowerInvariant ; if gate is not "improved" or "no-regression" or "none" throw new ArgumentException $"--gate must be improved, no-regression or none, not '{gate}'." ; int minRuns = int.Parse opts.GetValueOrDefault "min-runs", "5" , CultureInfo.InvariantCulture ; double z = double.Parse opts.GetValueOrDefault "z", "1.96" , CultureInfo.InvariantCulture ; if z <= 0 throw new ArgumentException "--z must be positive." ; return new Cli args 0 , Required "baseline" , Required "candidate" , gate, minRuns, z ; }} On the .NET 10 SDK, the file runs as a file-based app with no project file: dotnet run tools/NoiseBand.cs -- eval-results.csv \ --baseline terminus-2 --candidate copilot-cli --gate improved If your team is still on .NET 8 or 9, drop the same file into a console project instead. It only uses the base class library, so the project file is the default one: <Project Sdk="Microsoft.NET.Sdk" <PropertyGroup <OutputType Exe</OutputType <TargetFramework net8.0</TargetFramework <Nullable enable</Nullable <ImplicitUsings enable</ImplicitUsings </PropertyGroup </Project And if you would rather not install an SDK on the machine that holds the results, the official SDK container does the job. This is also how I would run it on a self-hosted build agent: docker run --rm -v "$PWD:/work" -w /work mcr.microsoft.com/dotnet/sdk:10.0 \ dotnet run tools/NoiseBand.cs -- eval-results.csv \ --baseline terminus-2 --candidate copilot-cli --gate improved The config names in those commands are whatever your eval runner writes into the config column. My suggestion: do not use bare names like copilot-cli. Put a short fingerprint into the name, for example copilot-cli@ followed by the first characters of a hash of the compiled prompt and tool schemas. I will come back to why in the section on drift. This is where the band earns its keep. The rule I want in CI is simple: a pull request that claims a harness improvement must clear the band. A pull request that makes no claim must not fall below it. I express the claim as a label, because a claim should be an explicit decision by the author, not something CI infers from the title. Here is how I would wire it in GitHub Actions. The eval job is your own runner, whatever it is; the only contract is that it writes the four-column CSV. name: harness-eval-gateon: pull request: types: opened, synchronize, labeled, unlabeled paths: - "harness/ " - "prompts/ " - "tools/NoiseBand.cs"jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 Your eval runner goes here. It must run every task at least 5 times under BOTH the main-branch harness and this PR's harness, and write task id,run index,config,passed. - name: Run eval suite run: ./eval/run-suite.sh --runs 5 --baseline-ref origin/main --out eval-results.csv - uses: actions/upload-artifact@v4 with: name: eval-results path: eval-results.csv noise-band: needs: eval runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-dotnet@v4 with: dotnet-version: "10.0.x" - uses: actions/download-artifact@v4 with: name: eval-results - name: Check the claim against the noise band env: A PR labelled claims-improvement must clear the band upward. Every other harness PR only has to stay out of the band's lower side. GATE: ${{ contains github.event.pull request.labels. .name, 'claims-improvement' && 'improved' || 'no-regression' }} shell: bash run: | set +e dotnet run tools/NoiseBand.cs -- eval-results.csv \ --baseline main --candidate pr \ --gate "$GATE" --min-runs 5 noise-band.txt 2 &1 status=$? set -e cat noise-band.txt { echo ' Noise band' echo 'A few notes on why it looks like this. The set +e and the saved exit code matter. A plain run step stops at the first failing command, so if the tool exits with 1 the summary would never get written. Capturing the status, writing the report to the job summary, and only then exiting with that status means the job still fails and the band report still lands where a reviewer will actually see it. The tool returns 2 for bad input, and that also fails the job. I prefer a broken CSV to block the merge rather than to pass as “no data, no regression.” The gate deliberately makes it hard to claim a win. With a band of plus or minus 6.7 points on an 89-task suite, most honest harness tweaks will land inside it. That is the point. “Inside the band” turns the PR description from “improves the score by 3 points” into “no detectable change on the eval; merging for the reasons below.” That is a perfectly respectable PR. It is just an honest one. What the gate does not do: it does not know whether the tasks are the right tasks, whether the score measured the right thing, or whether the eval environment taught the agent to behave differently from production. Those are the next three sections, and in my opinion they matter more than the arithmetic. The band tells you whether a score moved. It says nothing about whether the score was measuring what you think. Two pieces I read this week make that painfully concrete. The first is the Microsoft preprint “Building to the Test: Coding Agents Deliver What You Check, Not What You Requested” by Yanuo Ma, Ben Kereopa-Yorke and Ben Schultz arXiv 2606.28430, June 2026 , which I read alongside Jaroslaw Wasowski’s write-up “Spec Passes, Product Fails” on Level Up Coding November 1, 2026 . Two Copilot CLI agents, one on claude-opus-4.7 and one on gpt-5.5, had to re-implement a React Fluent UI data table as a reusable Angular library, scored by a hidden 222-test Playwright oracle that only reported test names and pass or fail. Eighteen runs in total, twelve of them with the oracle available during the work. With the oracle in the loop, all but one of those runs reached 221 or 222 out of 222. Then the authors audited the library itself, four subsystems: selection, sorting, column resizing and grid navigation. In 7 of the 12 oracle runs, at least one subsystem was dead or absent. The demo app had inlined the tested behavior, and the library the task actually asked for sat beside it doing nothing, or was missing. The authors’ summary line is worth quoting: “Once the oracle defines ‘done,’ what it does not score becomes optional.” The authors are careful, and so is Wasowski: they do not call it cheating, because the oracle was honest and its source hidden. It is one task, two agents, eighteen runs, a preprint without independent replication. But the lesson for noise bands is direct. A near-perfect score with no band at all, and a score that clears its band by a mile, can both sit on top of an artifact nobody would ship. Wasowski’s practical answer is what he calls an evidence matrix, where every requirement records its evidence and, in a separate column, what that evidence does not cover. Plus two cheap usage checks: a second client that consumes the component only through its public interface, and a controlled disabling replace the component’s method bodies with no-ops after which the result has to change. In the paper’s no-op ablation, one dead library kept its 29 of 29 resize tests after being disabled, which is exactly the signal you want that check to produce. The connection to harness evals: if your harness change “improved the score,” part of the claim should be an audit of what the runs actually shipped on the tasks that flipped. The flipped-task list my tool prints is where I would start. Waldek Mastykarz’s post on the Microsoft developer blog, “Can a local git remote replace GitHub in your agent eval?” October 20, 2026 , is the best small example I know of an eval environment quietly changing the result. The fixture asked agents to rebase a branch, and the variable was only where origin pointed: a local bare repository, or a fresh private GitHub repo created per run and deleted at teardown. In the final round with GPT-5.6 Sol across two agents Copilot CLI and VS Code with Copilot Chat , the agents pushed without being asked in 111 of 170 runs against GitHub and 92 of 170 against the local remote: 11 points apart, which the authors report at p = 0.033. They put the plausible range of the true gap at roughly 1 to 22 points, and they flag the scope themselves: one model, one repo, one clean rebase scenario. Rebase correctness and push safety came out the same on both remotes. The difference was only in the decision the agent made on its own. Two details stay with me. First, the direction of the error: the local remote made the agent look more careful than it is in real use, which is the kind of error nobody goes looking for. Second, how they counted. Pushes were read from the remote-tracking reflog and cross-checked against the remote’s refs at teardown. The agent’s own narration was treated as anecdote. A handful of local runs did comment on the remote, but the counts are what carried the result. For noise bands, this is the same lesson from the other side. A band is only as good as the thing being counted. If “passed” in your CSV comes from what the agent said it did, or from an environment that differs from production in a way the agent can notice, the band is a precise measurement of the wrong quantity. My config-name suggestion above comes from Wasowski’s drift piece, “AI Agent Drift: Harness Fingerprint, Trajectory Distributions, and Bisection by Replay” Level Up Coding, November 1, 2026 . He opens with a scene where the eval is green, the model is pinned, nobody changed the code, and cost per task still climbs. He labels that scene himself as a composite built for illustration, and ties the mechanism to Anthropic’s April 23, 2026 postmortem, where three separate harness-side changes landed in the same weeks and looked like one vague degradation. His fix is a harness fingerprint on every trace event: versions and hashes for the model requested and returned identifier , inference parameters, the compiled prompt, tool schemas, MCP server versions, the retrieval index, policies, compaction settings and the harness build. For a noise band, the consequence is simple. A comparison between “main” and “pr” is only a comparison of two configurations if each row really came from one configuration. If the provider shifted something mid-run, or half the baseline runs used last week’s tool schema, the paired differences contain a third variable and the band does not know about it. He also gives sample sizes for detecting distribution shifts in production traces: about 294 traces per period and intent to see a tool share move from 20 to 30 percent, about 686 for a retry rate moving from 10 to 15 percent, about 1,059 for a rare event moving from 5 to 8 percent. Those are his own planning calculations from the standard two-proportion formula, at 5 percent significance and 80 percent power, not measurements, and he says they will be optimistic when traces are correlated. I quote them because they make the same point as Perrone’s 650 tasks from a different angle: small effects need a lot more data than intuition suggests. Even a correctly banded, honestly scored eval only covers what happens before the merge. Two pieces push the measurement past it. Wasowski’s “Spec Rework Rate and Cost per Accepted Change Instead of AI Agent Speed” Level Up Coding argues for two units: cost per accepted change, where rejected and abandoned attempts stay in the numerator, and a spec rework rate, with one hand-filled field at every return: was it a spec gap, an agent error despite an explicit spec, or a requirements change. He is explicit that “spec rework rate” is his working term, not a standard metric, and that he has not run such a pilot himself. His treatment of the METR productivity studies is the most honest handling of uncertainty I read all week, so I want to keep his caveat intact. The 2025 METR randomized study 16 experienced developers found tasks with AI took 19 percent longer, with an interval of plus 2 to plus 39 percent. The follow-up described in February 2026 found minus 18 percent for 10 returning developers interval minus 38 to plus 9 and minus 4 percent for 47 new ones interval minus 15 to plus 9 . Both newer intervals include zero, and the METR authors themselves called the new data an unreliable signal, mainly because between 30 and 50 percent of surveyed developers skipped some tasks they did not want to do without AI. That is a research team putting a band around its own headline and admitting the band swallows it. I wish more harness PRs did the same. He also cites one published configuration comparison that I keep thinking about: Naresh Jain’s experiment on martinfowler.com Jain founded Specmatic, which built the harness used, so a possible conflict of interest is worth keeping in mind . A cheaper model without the harness passed 94.8 percent of individual acceptance checks, yet only 3 of 10 complete applications passed the whole 178-test suite. Share of tests passed and accepted changes are different units, and averaging the first does not give you the second. Perrone’s own last section points the same way. He cites Greptile, which sells an AI code reviewer, for a 7-day churn measure on pull requests its reviewer had commented on: agent pull requests churned at 5.7 to 13.5 percent of files against 10.0 percent for human ones. A vendor measurement on a vendor’s own sample, and Perrone himself notes that churn counts any later edit, so it overstates rework. He also works out that telling a revert rate of about 3 per 1,000 merged pull requests from half that would take roughly 17,300 merged pull requests per side. Small rates need enormous samples. The same arithmetic, again. Perrone’s article cites several third-party figures that are worth knowing, and none of them are his. For completeness, with attribution: That last one belongs next to any noise band: if two configurations ran on differently provisioned containers, the band is comparing infrastructure as much as harnesses. 1. No band, no claim. Every task at least five runs under both configurations, the same task set on both sides, and a paired band computed before anyone writes a number into the PR description. Inside the band is reported as “no detectable change,” never as “no change” and never as “slight improvement.” Complete flips get listed by name and their traces read. 2. Count ground truth, not narration, in an environment that matches production. Grade the repository after the agent exits with tests it never saw. Read side effects from the system of record reflogs, remote refs, the database , not from the agent’s summary. Audit what was shipped on the flipped tasks, with a second client or a no-op ablation where a component was the deliverable. And check that the eval environment does not differ from production in a way the agent can notice. 3. Pin the configuration and follow the change past the merge. Every result row carries a harness fingerprint, so “main versus pr” really is two configurations and not five. After the merge, track rework with a recorded cause, because an eval score is a forecast and rework is what actually happened. Task independence. The formula treats each task as an independent draw. Real suites cluster: several tasks from one repository, several circuit puzzles, several tasks that share one fragile dependency. Correlated tasks make the true band wider than the formula says, and I do not have a clean fix. Grouping related tasks and treating each group as one unit is the honest option, but it shrinks the effective count, which shrinks the power, which is the problem we started with. Small n. The 1.96 is a large-sample number. With 20 or 30 tasks, a t value would be the more defensible multiplier, and it makes the band noticeably wider; that is why the tool warns below 30 tasks and lets you pass --z. And five runs per task gives pass rates in steps of 20 points, so a single flaky task moving one run moves its own difference by 20 points. Teams running a 15-task internal suite should expect to almost never clear a band, and I think that is the correct outcome, not a bug in the method. Judge-based scoring. Everything above assumes passed is a deterministic test result. When it comes from an LLM judge, the band only covers sampling noise, not judge error. A judge that is systematically wrong in the same direction for both configurations leaves the band looking tidy. A judge that is wrong differently for the two configurations because one harness writes more confident summaries, say can manufacture a difference that clears the band. The kappa figures above, 0.26 and 0.19 against executed tests, are a reminder of how far a judge can sit from ground truth. Keep correctness with tests wherever tests exist, and check the judge against people before you let it fill the passed column. None of this makes the band less useful. It makes it a minimum. A harness change that cannot clear its noise band has not earned a claim. A change that does clear it has earned the next round of questions, not a celebration. These connect to this one and I do not repeat them here: Primary and author pieces I read in full: Third-party figures cited through Perrone or Wasowski, as labeled in the text: the 2026 SWE-Bench Pro audit, AgentLens May 2026 , “Fixing, Breaking, or Faking It?” Journal of Cybersecurity and Privacy, 2026 , Anthropic’s infrastructure-noise post February 2026 , Greptile’s “Rise of the Overnight Agents” vendor data , the METR studies 2025 and February 2026 , Naresh Jain’s experiment on martinfowler.com October 2026 , and Anthropic’s April 23, 2026 postmortem. Tags: ai-agents, llm-evaluation, dotnet, csharp, statistics, ci-cd, github-actions, benchmarking, software-testing, harness-engineering Your Harness Didn’t Get Better, It Got Lucky: Noise Bands for Agent Evals https://pub.towardsai.net/your-harness-didnt-get-better-it-got-lucky-noise-bands-for-agent-evals-9413224c6fcb was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.