The Eval Harness: 60 Local and Cloud Coding Runs

A 60-run evaluation of coding agents found no quality winner among local DeepSeek V4 Flash, hosted DeepSeek via Vercel AI Gateway, and Claude Sonnet 5, with strict acceptance at 8/20, 9/20, and 5/20 respectively, but local DeepSeek had a 20-minute median agent time and 13 deadline hits versus 8.7 minutes and $1.35 total for hosted DeepSeek and 12.4 minutes and $33.44 for Sonnet. The author, who maintains two repositories, ran the same agent shell (Claude Code) across all routes and found that one-to-three-file changes were feasible while four-plus-file changes dropped to 3/7 held-out-check passes on every route.

Model cards do not answer the question that actually matters to me: when real coding work lands in one of my own repositories, which inference route should I drive it through — the big local model on my desk, the same model rented from a cloud gateway, or a frontier model that costs more per run than the other two combined? The local route is the part I keep coming back to. The machine is a 128 GiB M5 Max, and resident in its unified memory is an 80.76 GiB quantization of DeepSeek V4 Flash — a frontier-class coding model that fills most of the machine’s RAM budget on its own. There is no network cord on that path and no API meter ticking against it. A few years ago a model this size lived in a rack in someone else’s building; here it sits on a desk, and an agent loop can hand it a real repository and ask it to ship a patch. The question is no longer whether that is possible. It is whether the patch is any good, and what it costs in waiting. I ran the same model three ways, with a frontier comparison alongside it. Local DeepSeek from the resident weights. Hosted DeepSeek through Vercel AI Gateway — the same model family served from cloud providers. And Claude Sonnet 5 as the frontier hosted lane. One agent shell, Claude Code, drove all three through an Anthropic-compatible endpoint, so the inference could move while everything around it — tools, prompt, fixture, grader, repair policy — stayed fixed. Then I gave that shell sixty isolated coding sessions to finish. The task set held twelve historical changes I had already made and accepted in two repositories I actively maintain; four of them ran three times per route to expose variance. Every agent started from the parent commit, with no remote and no future Git history. When an agent stopped, a grader copied its worktree, overlaid held-out target tests the agent had never seen, ran the declared commands, checked the patch boundary, and offered one repair pass when the first attempt had finished but failed. The headline scores were close, and none of them crowned a winner. Strict acceptance landed at 8/20 for local DeepSeek, 9/20 for hosted DeepSeek, and 5/20 for hosted Sonnet. Held-out executable checks passed in 12/20, 13/20, and 12/20. No paired route comparison established a quality winner. The operational gap was far wider than the quality gap: local DeepSeek had a 20-minute median agent time and hit an agent deadline in 13 of 20 runs; hosted DeepSeek had an 8.7-minute median and one timeout, for $1.35 total; Sonnet had a 12.4-minute median, made roughly twice as many model requests, and cost $33.44. What the sixty runs produced is less a ranking than a workload map. One- to three-file changes were reasonable candidates for these agents. Four-plus-file changes dropped to 3/7 held-out-check passes on every route. Sonnet completed all three repetitions of a narrow frontend concurrency fix, while every route failed every repetition of a broad sensitivity-control change. The right decision depends on patch shape, deadline cost, and review policy — not on a model name. The complete record has its own interactive flight recorder. It exposes every cell, timing mark, provider attempt, exact bill, scoring lens, and local-cost assumption behind this drawing. The decision came before the benchmark My recent OpenRouter versus Vercel AI Gateway comparison /blog/openrouter-vs-vercel-ai-gateway covered product design: price presentation, developer experience, model catalogs, routing controls, and provider redundancy. It left a harder question open. Product features describe what a gateway can do; they do not say whether hosted inference earns its bill on my code, whether a large local model can keep up, or whether a more expensive frontier model turns that extra cost into a better accepted patch. Public coding benchmarks provide candidate-screening evidence. SWE-bench, for example, asks a system to modify real repositories and applies repository tests to the resulting patch 1 . Transfer to my decision remained uncertain. My repositories use their own conventions, dependency graphs, half-finished migrations, front-end state, security boundaries, and build behavior. My agent shell, time budget, caching pattern, hardware, and review policy also shape the deployed system. The decision sentence was concrete: ctx and the portfolio, under one shared agent shell, with executable correctness, patch integrity, elapsed time, reliability, and cost visible on the same record.That sentence ruled out a collection of toy functions. I mined changes I had already made and accepted in the repositories, then rebuilt each task from its parent commit. The historical target patch was never shown to the agent. It served two purposes before the matrix began: derive independently authored target checks, and prove that each grader rejected the parent while accepting the historical change. The evaluated unit was the complete agent route. A model request in isolation cannot edit a file, inspect a repository, run a command, recover from a failed test, or stop before a deadline. Claude Code 2.1.221 supplied the agent loop and tools. A preflight environment snapshot had recorded 2.1.220 several hours earlier. Before the run plan froze, I pinned the 2.1.221 binary by path, reported version, and SHA-256; every scheduled cell launched that binary. The inference route supplied the model responses. The fixture, prompt, time limits, grader, and repair policy completed the instrument. Vercel’s Anthropic Messages endpoint made the shared shell possible: Claude Code can point at the Gateway with an ANTHROPIC BASE URL change 2 . Anthropic documents gateways as a central proxy layer for authentication, usage tracking, budgets, audit logs, and model routing . For the local lane I supplied the same protocol from a relay in front of the resident model. Protocol compatibility held the agent constant while inference moved. 3 ref-anthropic-gateway-config The harness around that loop was its own build. I used Codex to stand up the scaffolding and the test harness — the fixture builder, the grader, the scheduler, and the publication pipeline. Keeping that work off the Claude Code agent under test meant the instrument was never measuring itself. With the run plan frozen and the binary pinned, the matrix did not need anyone at the keyboard: it launched that evening and ran unattended through the night, finishing all sixty cells near noon the next day. Twelve accepted changes became the task population The suite covered two repositories and several failure modes. Eight tasks came from ctx , my personal context engine: canonical UUID validation, a shared upstream timeout, public model-catalog parsing, catalog filtering, callable Gateway slugs, race-safe credential reads, a fail-closed sensitivity screen, and OAuth/connector hardening. Four came from this portfolio: canonical inquiry labels, newsletter conversion state, concurrent submit deduplication, and a server-decided editorial rail. Each prompt specified desired behavior at the detail level of a well-written engineering ticket. Several named exact mappings or validation shapes. The suite therefore measures scoped implementation work; vague issue triage remains outside its claim. Both source repositories are public. All twelve target commits were created on August 2–3, 2026, after DeepSeek V4’s April 23 release 4 and Sonnet 5’s June 30 release . The exact target diffs could not have appeared in either model’s pre-release training corpus, and the isolated agents had no network, remote, or future Git history. Earlier public code and repository conventions may still have appeared in training, so repository familiarity remains a possible source of lift in the acceptance rates. 5 ref-anthropic-sonnet-5 The expected patch footprint supplied a useful pre-run slice. Nine route-level cells expected one changed file, four expected two or three, and seven expected four or more. Those counts are per route after adding repetitions. The task families included validation, parsing, backend routing, reliability, OAuth, security, concurrency, full-stack state, and front-end architecture. Four anchors received two extra repetitions on every route: - canonical UUID validation: a narrow one-file validation change; - race-safe credential reads: a four-file concurrency and security change; - fail-closed sensitivity screening: a four-file security and model-integration change; - concurrent submit deduplication: a one-file front-end concurrency change. This yielded twenty cells per route: twelve first repetitions plus eight extra anchor repetitions. Sixty runs were enough to expose recurring behavior, while still leaving wide statistical intervals. One anchor was easy and stable for local DeepSeek. One was consistently impossible for every route. The other two exposed route-specific variance. Every fixture was a new repository with no remote. Dependency state was prepared in advance. The model saw a task prompt and the repository at its parent commit. Each run received a fresh agent home and relay log. This prevented one model’s edits, session memory, or Git history from entering the next cell. The local and hosted DeepSeek lanes share a model family and revision label, but they do not represent identical numerical inference. The local file was an IQ2XXS mixed quant with Q8 projections and an 80.76 GiB footprint, pinned by SHA-256. The Gateway lane could use different precision, kernels, batching, and serving providers. I treat that pair as local versus hosted delivery of the same model family, rather than a pure hardware comparison. Context was another route-level difference. The local runtime was pinned to a 131,072-token ceiling; provider-controlled Gateway ceilings were not frozen in the manifest. The largest per-response input and cache accounting totals were 70,112 tokens locally, 102,907 for hosted DeepSeek, and 169,109 for Sonnet. The local Anthropic-compatible runtime recorded fresh prompt tokens under cache creation and reused context under cache reads, leaving the plain input-token field at zero. The combined accounting above captures that prompt volume. The local record stayed below its configured limit, while context ceilings and agent compaction behavior remain part of the complete-route comparison. The grader kept executable behavior and patch discipline separate Each case carried hidden files, hidden commands, visible repository commands, an expected path list, and a known historical patch. Validation happened before the paid matrix. The unchanged parent had to fail its hidden checks. Applying the historical patch had to pass the complete executable command set. Reference validation skipped patch integrity because accepted historical changes may include their own tracked-test edits; the integrity rule applied to agent output. A case that failed either executable validation never entered the run plan. Eleven cases used historical held-out targets. The portfolio editorial-rail case used a hidden oracle I authored for the benchmark, then froze and validated before the paid matrix and before any agent output existed. After an agent pass, the runner captured Git status and copied the worktree into a grader directory. Hidden target tests were then overlaid into that copy. The agent never saw those test files while editing, and any test changes made in its worktree could not alter the overlaid target checks. The grader ran hidden commands followed by declared validation commands, retaining exit codes, timeouts, and output tails. The two result lenses answer different questions: Held-out checks pass means every required command completed successfully after the target-test overlay. Strict acceptance adds the integrity rule: tracked tests, lockfiles, and Git metadata must remain unchanged. The split became necessary because several agents produced functionally correct patches while adding or updating tracked tests. Those changes did not weaken the hidden target checks; the overlay made that route unavailable. They still crossed the benchmark’s patch boundary. Counting both outcomes lets a reader distinguish implementation behavior from instruction and repository-discipline behavior. The first attempt had a twenty-minute wall-clock limit. At the deadline, the runner terminated the process and graded the captured patch. A finished but rejected attempt could receive one ten-minute repair in the same session. Repair feedback contained failed command labels and bounded output tails, plus an explicit instruction to leave existing tests alone. A process timeout was final because a resumed agent after forced termination would be a different recovery policy. Required commands had their own fifteen-minute ceiling. Upstream first-byte and stream-idle limits were each two minutes. Repairs helped, but modestly. Local DeepSeek used six repairs and converted two to strict acceptance. Hosted DeepSeek used fifteen and converted five. Sonnet used seventeen and converted three. A repair opportunity therefore belonged in the measurement record; it could not be treated as guaranteed recovery. First-attempt counts include R10, whose passing patch was captured at the 20-minute deadline. Local API spend excludes hardware and electricity; neither was measured. 5/9 3/4 1/7 This lens rejects patches that changed tracked tests, lockfiles, or Git metadata, even when every overlaid target check passed. - ▸Decision and route contract - ▸Historical-task and anchor inventory - ▸Fixture validation and hidden-scorer plan - ▸Per-run evidence and recovery ledger - ▸Paired analysis and workload-routing worksheet The quality ranking remained unresolved Strict acceptance ended at 40% for local DeepSeek, 45% for hosted DeepSeek, and 25% for hosted Sonnet. Wilson 95% intervals were 21.9–61.3%, 25.8–65.8%, and 11.2–46.9%. Brown, Cai, and DasGupta show why the familiar symmetric normal interval behaves poorly for binomial proportions at small sample sizes; the Wilson interval has better coverage behavior across much of that range 10 . This calculation treats the twenty route cells as independent Bernoulli observations. The suite contains twelve distinct tasks and only four repeated anchors, so it does not model task-level correlation; the intervals are descriptive bounds for this matrix. Held-out-check rates were 60%, 65%, and 60%. The same twenty matched cells also permit paired comparisons. For strict acceptance, the exact two-sided paired p-values were 1.00 for local versus hosted DeepSeek, 0.453 for local DeepSeek versus Sonnet, and 0.219 for hosted DeepSeek versus Sonnet. For held-out checks, all three were 1.00. These p-values are unadjusted across the three route comparisons. Since none approached rejection, a Holm or Bonferroni correction would only increase the adjusted values and leave the null result unchanged. A future confirmatory matrix should predeclare its multiplicity rule. These data support no route-level quality winner. First-attempt strict acceptance tells another part of the story: 6/20 local, 4/20 hosted DeepSeek, and 2/20 Sonnet. The held-out-check first-attempt counts were 10/20, 9/20, and 7/20. “First attempt” describes the pre-repair patch, including a patch captured at forced termination. Local run R10 reached the twenty-minute deadline and still passed both scoring lenses; completed-within-cap local counts are therefore 5/20 strict and 9/20 held-out. The other routes’ first-attempt passes all completed within the cap. A model with more requests and a larger bill did not translate that extra search into a higher aggregate pass rate here. The integrity-only rejections were 4, 4, and 7. Sonnet’s strict score suffered most from this boundary. The distinction matters for use. A developer who reviews every patch and welcomes new tests may care more about the 12/20 held-out result. A protected branch that rejects any test-file change should use 5/20. Repeated anchors also constrain the story. Local DeepSeek passed canonical UUID validation in all three runs. Sonnet passed concurrent submit deduplication in all three after repair. Every route passed all three race-safe credential-read checks at the executable layer, yet strict integrity accepted only one hosted DeepSeek run. Every route failed all three sensitivity-screen runs. Model names alone hide these workload shapes. Hosted DeepSeek bought back eleven minutes per task The local route charged no API bill. It also produced the largest operational cost in the experiment: waiting. Median agent time was 20.0 minutes, with 13/20 runs reaching a first-attempt or repair deadline. Hosted DeepSeek’s median was 8.7 minutes with one timeout. Sonnet’s median was 12.4 minutes with three agent timeouts. Hosted DeepSeek used 666 model requests, 33.3 per run. Sonnet used 1,340, exactly 67 per run on average. Local DeepSeek used 583, or 29.15 per run. The Sonnet route therefore spent more turns searching, reading, editing, and testing without raising the aggregate held-out-check count beyond the other lanes. The exact Gateway bill was $34.793258906: $1.352117006 for hosted DeepSeek and $33.4411419 for Sonnet. That works out to 6.8 cents versus $1.67 per run. Cost per held-out-check pass was about 10 cents for hosted DeepSeek and $2.79 for Sonnet. Cost per strict acceptance was 15 cents and $6.69. These ratios are descriptive point estimates with small success counts. Moving the denominator by one changes the held-out figures to 9.7–11.3 cents for hosted DeepSeek and $2.57–$3.04 for Sonnet; the strict figures move to 13.5–16.9 cents and $5.57–$8.36. The run plan froze a list-price estimator before the first cell. It predicted $1.291 for hosted DeepSeek and $50.162 for Sonnet. Actual response metadata landed 4.7% above the DeepSeek estimate and exactly one-third below the Sonnet estimate. The article and explorer use billed metadata. Vercel states that AI Gateway rates follow provider list prices with no token markup 8 ; provider, caching, and the rates attached to the completed request still determine the final ledger. Local’s zero remains an API-cost statement. I could not collect joule measurements because powermetrics required an interactive administrator credential. The run log recorded that limitation and disabled energy claims. Hardware price, useful life, inference allocation, power draw, electricity rate, and utilization remain scenario inputs. The evidence explorer’s calculator /demos/local-vs-cloud-ai-coding calculator-title lets readers supply them. A future drawing will measure the local inference ledger directly. Provider and credential recovery appeared in the request trace Hosted DeepSeek produced 663 successful model requests. Fireworks completed 411, Alibaba 136, Baseten 106, and Novita 10. Almost half of those successful responses—323, or 48.7%—included at least one failed provider attempt first. The trace recorded 479 failed attempts: 300 Baseten 503s, 169 Fireworks 503s, and 10 Alibaba 503s. Only three model requests failed outright, all at the experiment’s two-minute first-byte limit. That is measured redundancy. Vercel documents default provider choice as a dynamic decision based on recent uptime and latency, with explicit ordering, filtering, caching, and timeout controls available to the caller 6 . The matrix used the default provider pool and retained the attempt metadata. A single requested model reached four serving providers while the agent continued through hundreds of upstream failures. Sonnet exercised a different recovery layer. Every one of its 1,325 successful requests first tried an Anthropic BYOK credential with depleted balance, failed in a median 160 milliseconds, then completed on Vercel’s managed Anthropic credential. Vercel’s BYOK policy describes this exact fallback from a failed customer credential to system credentials 7 . Six later model failures were local per-run budget stops; nine were first-byte or stream-idle timeouts. Provider fallback and model fallback are separate controls. The former keeps a requested model and changes the host. The latter moves to a backup model after every provider for the primary has failed; Vercel applies provider routing inside each model before advancing through the model list 9 . This matrix measured provider fallback and credential fallback. It did not configure cross-model fallback, because changing the requested model mid-cell would break the route comparison. Patch footprint predicted the hard boundary better than model price Expected scope produced the clearest slice. On one-file tasks, held-out checks passed 6/9 for local DeepSeek, 6/9 for hosted DeepSeek, and 7/9 for Sonnet. On two- to three-file tasks the counts were 3/4, 4/4, and 2/4. At four or more expected files, every route fell to 3/7. Treat these small slices as directional trial assignments rather than population estimates. Each anchor had three repetitions, and each scope denominator held four to nine cells per route. Equal aggregate counts also do not establish numerical equivalence between the local quant and hosted model; quantization and serving infrastructure move together in that comparison. The route recommendations follow those measured boundaries: Local DeepSeek fits non-urgent, narrow work when the machine is already available and API spend matters more than elapsed time. It completed all three UUID repetitions and matched the cloud lanes on aggregate executable checks. Its timeout rate requires a queue, patience, and a human willing to resume or take over. Hosted DeepSeek is the default for routine one- to three-file work in these repositories. It matched local quality, cut the median agent time by more than half, and cost seven cents per run. It was the only route to pass the OAuth hardening task, and the only route to reach 4/4 held-out passes in the two- to three-file slice. Hosted Sonnet earned a targeted role rather than a general promotion. It reached 3/3 on concurrent submit deduplication and 7/9 on one-file held-out checks. Its broader aggregate, request count, and bill supported no case for using it on every task in this suite. Human-led decomposition fits broad security and architecture work. The four-file sensitivity change failed 0/3 on every route. The six-file editorial rail, which used the suite’s one benchmark-authored oracle, failed all three single trials. Breaking those jobs into independently testable changes is better supported by the data than switching to the most expensive model. These are allocation rules for this task distribution, agent version, model configuration, date, and budget. The suite has twelve distinct tasks and only four repeated anchors. Route intervals remain wide. Repository-specific checks may reward conventions already common in the models’ training data. Local quantization and hosted precision differ. A different agent shell can alter search behavior and tool use. A future matrix should add more repositories, more repeated tasks, direct energy instrumentation, and route-blinded human patch review. Every cell retained enough evidence to be challenged The matrix ran from 01:30:52Z to 16:09:47Z on August 4, 2026 — 14 hours, 38 minutes, and 54 seconds, unattended from the first cell to the last. Each cell wrote to its own run directory as work progressed. The durable record included the prepared fixture, isolated agent home, first-attempt event stream, repair event stream when used, relay JSONL, first and repair grades, command logs, captured patch, start and end machine samples, cost summary, provider attempts, and final run JSON. A failure in a later cell could not erase an earlier cell. The final audit reconciled 60 run records with 60 relay logs. All endpoint machine samples reported AC power. One APFS fixture copy failed before inference began, spent $0, and was moved under an excluded directory with its evidence intact. Its replacement completed the missing matrix cell. Exclusions therefore remain visible without entering the result denominator. The matrix lock names the pre-run harness commit. Later commits added analysis and review tooling, so lock verification reads the original Git blobs at that pinned commit. All 39 locked file hashes, the manifest hash, and the schedule hash matched. Raw traces stay private because they contain prompts, model events, local paths, session identifiers, and provider request IDs. A deterministic analysis script produces two checked-in publication artifacts. The aggregate file contains route, task, slice, latency, failure, routing, and cost statistics. The explorer file adds one sanitized row per run while excluding response bodies, credentials, local paths, and private identifiers. The interactive flight recorder /demos/local-vs-cloud-ai-coding is the publication surface for that sanitized artifact. It supports several readings of the same work: - switch between strict acceptance and held-out checks; - inspect all sixty cells by task, route, and repetition; - select a run to see patch size, requests, repair time, provider, cache reads, and exact bill; - compare the duration distribution rather than a single mean; - trace failed providers and credential recovery; - follow cumulative spend and accepted-patch marks; - examine repository and expected-scope slices; - enter hardware and electricity assumptions without presenting them as measured energy. That separation keeps the article readable while leaving the evidence open to inspection. A reader can reject my integrity policy, choose the executable lens, and watch the route totals change. They can treat timeouts as a deployment failure, or focus on completed checks. They can price the local machine against their own workload instead of inheriting my hardware allocation. The experiment changed my default. Hosted DeepSeek is now the first route I would choose for bounded repository work under time pressure. Local DeepSeek remains viable for queued narrow tasks on hardware I already own. Sonnet gets a targeted trial when the task resembles the front-end concurrency anchor it handled consistently. Broad security and architecture changes get decomposed before any agent starts. That is as far as sixty runs can responsibly carry the claim. The experiment found useful operational differences and a workload boundary. It did not establish a universal model ranking. The next drawing will extend the local side with measured power, utilization, amortization, and break-even curves so “$0 API” can become an honest total-cost line. - 01 Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?,” arXiv:2310.06770v3, 2024 https://arxiv.org/abs/2310.06770 ↩ cite-jimenez-2024-swebench - 02 Vercel, “Anthropic Messages API,” AI Gateway documentation, updated March 7, 2026 https://vercel.com/docs/ai-gateway/sdks-and-apis/anthropic-messages-api ↩ cite-vercel-anthropic-messages - 03 Anthropic, “LLM gateway configuration,” Claude Code documentation https://docs.anthropic.com/en/docs/claude-code/llm-gateway ↩ cite-anthropic-gateway-config - 04 Vercel, “DeepSeek V4 on AI Gateway,” April 23, 2026 https://vercel.com/changelog/deepseek-v4-on-ai-gateway ↩ cite-vercel-deepseek-v4 - 05 Anthropic, “Introducing Claude Sonnet 5,” June 30, 2026 https://www.anthropic.com/news/claude-sonnet-5 ↩ cite-anthropic-sonnet-5 - 06 Vercel, “Provider Options,” AI Gateway documentation, updated February 9, 2026 https://vercel.com/docs/ai-gateway/models-and-providers/provider-options ↩ cite-vercel-provider-options - 07 Vercel, “Bring Your Own Key BYOK ,” AI Gateway documentation, updated January 21, 2026 https://vercel.com/docs/ai-gateway/authentication-and-byok/byok ↩ cite-vercel-byok - 08 Vercel, “AI Gateway Pricing,” updated February 10, 2026 https://vercel.com/docs/ai-gateway/pricing ↩ cite-vercel-pricing - 09 Vercel, “Model Fallbacks,” AI Gateway documentation, updated January 30, 2026 https://vercel.com/docs/ai-gateway/models-and-providers/model-fallbacks ↩ cite-vercel-model-fallbacks - 10 Brown, Cai, and DasGupta, “Interval Estimation for a Binomial Proportion,” Statistical Science 16 2 , 2001 https://www.stat.purdue.edu/~dasgupta/publications/tr99-19.pdf ↩ cite-brown-binomial-2001 Anything on this sheet still unclear — or anything you were too polite to ask out loud? File an RFI. Answers come from the drawing itself and cite their sheet numbers, and every question is recorded in the drawing log so the next revision can answer it in print.