{"slug": "the-harness-is-not-the-model-how-far-scaffolding-takes-a-weak-llm", "title": "The Harness Is Not the Model: How Far Scaffolding Takes a Weak LLM", "summary": "An engineering analysis of LLM scaffolding reveals that a strong harness—comprising retrieval, curated tools, and verification loops—can close the performance gap on knowledge and narrow verifiable tasks, buying roughly one model generation of improvement. However, the harness hits a model-bound ceiling on judgment, multi-turn coherence, and long-horizon autonomy, where model size dominates. The evidence supports tiered routing with frontier escalation as the optimal architecture.", "body_md": "## Abstract\n\nI made an engineering bet: wrap a swappable weak model (local on a Laptop, or a cheaper API tier) in the strongest possible harness, deterministic retrieval from a personal knowledge base, a curated toolset, verification loops, and expect frontier-level behavior from the assembly. This paper formalizes that bet into five testable components and tests each against roughly 30 published sources from 2022 through mid-2026. The evidence splits cleanly. The harness closes the gap almost fully on knowledge (an 11B retrieval-augmented model beating a 540B model on NaturalQuestions), on single-turn tool-call syntax, and on narrow, cheaply verifiable tasks (a roughly 15x swing on SWE-bench with weights held constant), buying roughly one model generation, a 2 to 6x multiplier from a naive baseline. It then hits a model-bound ceiling: same-harness model swaps produce 1.5 to 4x spreads on agentic coding and 50 to 100x spreads on METR autonomous time horizons, multi-turn tool coherence collapses with model size, and self-correction without an oracle degrades output. I concede the components that fail, keep the ones that survive, and end with the architecture the evidence supports: tiered routing with frontier escalation, and a rule of thumb that once a competent harness exists, swapping the model dominates every other intervention.\n\n**Keywords:** harness engineering, LLM scaffolding, test-time compute, agentic coding, function calling, local inference, tiered routing, model cascades.\n\n## 1. Introduction\n\n### 1.1 The Bet\n\nThe plan was concrete. I maintain a personal knowledge base, a vectorless Markdown wiki with deterministic two-step retrieval, and I wanted to connect it to a swappable local model running on a Laptop. The brain supplies the facts. A curated toolset supplies the hands. A harness (structured prompts, staged workflows, verification gates, retry logic) supplies the discipline. The model in the middle becomes a commodity: swap in whatever runs on the hardware this month, and the assembly around it produces frontier-level thinking and execution anyway. If the bet held, I would own the whole stack, pay no per-token rent for routine work, and upgrade by config change.\n\nCall this position harness maximalism. I believed a strong version of it, and my own systems gave me reasons to. I had watched a hierarchical multi-agent setup cut per-module delivery time by roughly 55 percent across more than 48 measured sessions, and I had watched a Haiku-grade prompt-writer model, given a deliberately narrowed task, cut sub-agent token consumption by 40 to 60 percent and drop retries from 1.5 to 0.6 per task. Scaffolding visibly multiplied output. It was tempting to extrapolate: if structure buys this much at the top tier, surely enough structure lifts the bottom tier to the top.\n\nThis paper is the test of that extrapolation. I went looking for published evidence that would either confirm the bet or break it, and I report both outcomes honestly.\n\n### 1.2 Formalizing the Bet\n\n\"Frontier-level performance\" is not one claim. It bundles capability dimensions that behave differently under scaffolding, so the bet decomposes into five testable components:\n\n**Knowledge**: retrieval substitutes for what the model knows.\n\n**Format reliability**: schemas and tuning substitute for native instruction-following on tool syntax.\n\n**Narrow verifiable tasks**: scaffolding plus verification substitutes for capability wherever an automatic check exists.\n\n**Judgment and coherence**: harness structure substitutes for the model's ability to hold state and reason across many turns.\n\n**Long-horizon autonomy**: orchestration substitutes for the model's ability to sustain a multi-hour task.\n\nTable 1 states the verdicts up front; the evidence draws the line between component 3 and component 4.\n\n**Table 1.** The closability matrix: each component of the bet, the harness element that bears on it most, and the verdict the published evidence supports.\n\n| # | Component of the bet | Primary harness element | Verdict | Anchor evidence |\n|---|---|---|---|---|\n| 1 | Knowledge | Retrieval | Closes, almost fully | Atlas 11B beats PaLM 540B (Izacard et al. 2022) |\n| 2 | Format reliability | Schemas plus tuning | Closes, single-turn only | ToolACE-8B above GPT-4-class on BFCL single-turn (BFCL 2026) |\n| 3 | Narrow verifiable tasks | Verification loops | Multiplies enormously, given an oracle | Roughly 2 percent to 33.2 percent, same weights (Yang et al. 2024; OpenAI 2024) |\n| 4 | Judgment and multi-turn coherence | Orchestration | Model-bound; decomposition mitigates, does not close | BFCL multi-turn collapse by model size (TinyLLM Survey 2025) |\n| 5 | Long-horizon autonomy | Orchestration | Model-bound; the widest gap of all | 50 to 100x METR time-horizon spread (METR 2025; METR 2026) |\n| all | Cost | Routing | Closes almost completely | 98 percent cost reduction at matched accuracy (Chen et al. 2023) |\n\nSections 2 and 3 walk the cells. Section 4 grounds them in what mid-2026 laptop hardware actually runs. Section 5 adds what I have measured in my own systems. Section 6 gives the architecture that survives, and Section 7 states the evidence caveats.\n\n### 1.3 Method and Scope\n\nThis is a synthesis of published evidence plus practitioner experience, not a report of new experiments. The 2022 to 2025 findings I rely on are primary-source papers. The mid-2026 leaderboard and throughput figures come from third-party aggregators; I flag them at first use and Section 7 states the resulting error bars explicitly. Every external number in this paper traces to a source in the References; the practitioner numbers in Section 5 are my own operational measurements, reported as such in Section 7.\n\n## 2. Where the Bet Holds\n\n### 2.1 Knowledge: Retrieval Substitutes Almost Fully\n\nThe strongest pillar under the bet is the oldest result. Atlas (Izacard et al. 2022), an 11B retrieval-augmented model, beat PaLM 540B on NaturalQuestions (above 42 percent at 64-shot, roughly 3 points ahead) and reached 84.7 percent on TriviaQA. That erases a roughly 50x parameter gap on knowledge tasks. What the model knows stops mattering when the harness hands it the right page; what it can do with the retrieved page is what remains.\n\nThis is precisely the component my knowledge base was built for. Its deterministic two-step lookup (a master index routes to 3 to 5 specific pages), its grep-anchored fact blocks, and its index discriminators were all designed to keep retrieval deterministic under weak-model conditions. The retrieval half of the bet degrades gracefully as the model shrinks. I keep this component in full.\n\n### 2.2 Format Reliability: Single-Turn Tool Syntax Is Closable\n\nOn the Berkeley Function-Calling Leaderboard, ToolACE-8B surpasses GPT-4-class models on single-turn function-calling accuracy (BFCL 2026). Strict schemas plus targeted tuning make an 8B model emit correct tool calls. The open-weight ecosystem confirms it at larger scales: GLM-4.5 tops BFCL v3 at 76.7 to 77.8 percent and Qwen3-32B posts 75.7 percent, ahead of closed API models on that table (BFCL 2026; marc0.dev 2026). Practitioner coverage agrees: for local agents, tool-calling reliability, not parameter count, decides whether a machine is useful at all (XDA Developers 2026). Single-turn function calling is, for practical purposes, solved locally. The failure lives one level up, in knowing when not to call and in carrying state across turns, and Section 3.2 returns to it. Component 2 survives, with an asterisk that matters.\n\n### 2.3 Narrow, Verifiable Tasks: The Multiplier Is Real and Large\n\nThe scaffolding literature's headline result is a natural experiment with the weights held constant. The same GPT-4-class model moved from roughly 2 percent on the original SWE-bench RAG baseline, to 12.47 percent under SWE-agent (Yang et al. 2024), whose interface-design ablation alone was worth 10.7 points, to 33.2 percent under the Agentless scaffold on SWE-bench Verified (OpenAI 2024). That is a roughly 15x swing from harness design alone. When I first assembled these numbers, the bet looked won: this is the claim, measured.\n\nRepeated sampling compounds it where a real verifier exists. Large Language Monkeys (Brown et al. 2024) took DeepSeek-Coder-V2 from 15.9 percent single-sample to 56 percent coverage at 250 samples on SWE-bench Lite, beating the then-frontier single-shot SOTA of 43 percent, because the test suite acted as ground truth. A mid-tier model plus an oracle-grade verifier plus compute outperformed the frontier model without them.\n\n### 2.4 Test-Time Compute Substitutes for Scale, Conditionally\n\nSnell et al. (2024) show a smaller model with compute-optimal test-time scaling (revisions plus a process verifier) outperforming a 14x larger model on MATH, with the compute-optimal allocation about 4x more efficient than naive best-of-N. The built-in caveat is load-bearing: this holds only on problems where the small model attains non-trivial success rates on its own. Test-time compute amplifies capability that already exists at some low rate. It does not create capability from zero, a boundary that reappears in Section 3.5.\n\n### 2.5 Cost Closes Almost Completely\n\nFrugalGPT-style cascades (Chen et al. 2023) match GPT-4 accuracy at up to 98 percent cost reduction by routing easy queries down-tier. But note what the cascade keeps: the frontier model, as the backstop for hard queries. Routing is a cost architecture, not a substitute for the top tier. This result quietly reframes the whole bet, and Section 6 builds on it.\n\n## 3. Where the Bet Fails\n\n### 3.1 Same Harness, Different Model: The Spread Is the Story\n\nThe cleanest way to test whether the harness or the model carries the performance is to hold the harness constant and swap the model. On SWE-bench Verified with scaffolding held constant, OpenHands CodeAct v3 scores 68.4 percent with Claude Opus 4.6 against 44.7 percent with GPT-5.2, and the observed range across model swaps runs 14.7 percent to 72 percent (CodeAnt AI 2026; aggregator-compiled figures, see Section 7). Sharper still: mini-swe-agent, a deliberately minimal 100-line ReAct loop over plain bash, exceeds 74 percent with a frontier model, beating far heavier harnesses (SWE-agent Team 2025).\n\nThat last result stung, because it inverts the bet's premise. Harness returns compress as the model strengthens; at the frontier, the best harness is nearly a no-op. Most of the headroom lives in the weights. The same-harness spread on agentic coding runs 1.5 to 4x, and no amount of scaffold sophistication on the weak side of that spread closes it.\n\n### 3.2 Multi-Turn Coherence Collapses with Model Size\n\nThe asterisk from Section 2.2 now becomes the finding. BFCL multi-turn data compiled in the TinyLLM survey (TinyLLM Survey 2025) shows Claude-3.5-Sonnet-class models near 90 percent overall while Qwen3-4B falls to 35.3 percent on multi-turn, Qwen3-1.7B to 16.9 percent, and Qwen3-0.6B to 1.4 percent. Single-turn schema compliance is harnessable; carrying state across turns is not, and no scaffold in the surveyed literature has fixed it. Component 4 of the bet fails on the published evidence.\n\n### 3.3 Long-Horizon Autonomy Is the Widest Gap of All\n\nMETR measures the length of task a model can complete autonomously at 50 percent success, in broadly comparable agentic harnesses (METR 2025; METR 2026). The progression: GPT-4o roughly 7 minutes, Claude 3.7 Sonnet roughly 50 minutes, o3 roughly 2 hours, Opus 4.6 and GPT-5.2 roughly 5 to 6 hours, Claude Mythos Preview 16 or more hours. That is a 50 to 100x spread between a 2024 mid-tier model and a 2026 frontier model, doubling roughly every 105 to 210 days, and it is model-bound. No known scaffold turns a 7-minute-horizon model into an hours-horizon agent. Component 5 fails outright.\n\n### 3.4 Self-Correction Without Ground Truth Is Capability-Negative\n\nThe reflex fix for a weak model is a reflection loop: have it check its own work. Huang et al. (2024) tested intrinsic self-correction without external ground truth and found it degrades reasoning, GPT-4 on GSM8K falling from 95.5 percent to 91.5 percent to 89.0 percent across correction rounds. Reflection loops pay off only when an oracle-grade verifier exists to stop them. This is scaffolding-shaped but capability-negative, and I had to remove it from my mental toolkit.\n\n### 3.5 Verification Saturates Without an Oracle\n\nBest-of-N sampling is bounded above by coverage, and coverage is bounded by whether the base model can ever emit a correct candidate. In domains without automatic verifiers, majority voting and reward models plateau after a few hundred samples (Brown et al. 2024). Reliability under repetition is its own axis: tau-bench's pass^k metric, which requires all k trials to succeed, collapses for small models that looked passable at pass@1 (Sierra Research 2024). And the Holistic Agent Leaderboard (HAL 2025), across 21,730 rollouts spanning 9 models and 9 benchmarks, found harness-knob effects non-monotonic, with higher reasoning settings reducing accuracy in 21 of 36 runs. Harness sophistication is not a dial that converts to model quality. Sometimes turning it up makes things worse.\n\n## 4. The Laptop Reality, Mid-2026\n\nThe bet named specific hardware, so the evidence review has to name what that hardware runs. The figures here are mid-2026 and largely aggregator-sourced; Section 7 sets the error bars.\n\n### 4.1 The Workhorse Class Is Sparse MoE\n\nOn 36 to 128GB Apple Silicon, the practical class is sparse mixture-of-experts models with roughly 3B active parameters. Qwen3-30B-A3B runs near 100 tokens per second on an M4 Max via MLX (WillItRunAI 2026; MarkAICode 2026). Qwen3-Coder-Next (80B total, 3B active, 256K context) runs 15 to 40 tokens per second depending on memory tier (Sienna 2026). gpt-oss-120b (5.1B active parameters, roughly 61 to 65GB in memory) needs the 128GB tier and returns 40 to 50 tokens per second (OpenAI 2025; Tenten 2026). Dense 70B models fell out of favor at 12 to 18 tokens per second; the mid-2026 comparisons treat the sparse class as the deployment default (AI.rs 2026).\n\n### 4.2 The Realistic Ceiling Is Sonnet-Class, About 12 Months Behind Frontier\n\nLaptop-runnable Qwen3-Coder-Next scores 42.8 percent on SWE-bench Verified against 79.6 to 88.7 percent for frontier closed models, while needing more agent turns (roughly 150 versus roughly 120) to get there (LLMCheck 2026; marc0.dev 2026). gpt-oss-120b reaches 80.1 percent on GPQA Diamond, o4-mini parity, against 88 to 94 at the frontier. Math is essentially closed: AIME 2025 with tools lands at 97.9 to 98.7 percent. The top open-weight models that do beat Sonnet-class (GLM-5 at 77.8 percent and Qwen3.5 at 76.4 percent on SWE-bench Verified) are 350B to 1T MoE that do not fit a laptop.\n\n### 4.3 The Walls: Prefill, Integration, Quantization\n\nPrefill, not decode, is the Apple Silicon wall. Unified memory bandwidth handles generation fine, but compute-limited prompt processing makes 100K-plus context agentic loops painful, which is exactly the workload a brain-connected assistant produces. Practitioners running Claude Code against local backends confirm the pairing works (Qwen 3.5-35B-A3B on 24GB and larger machines) but hit integration cliffs, including a KV-cache invalidation bug from per-request headers that cut throughput by roughly 90 percent until patched (KDnuggets 2026; Kibotu 2026). Community threads on local gpt-oss-120b report the same texture: the limits show up in use, not on the spec sheet (Hacker News 2025).\n\nQuantization, at least, is nearly free. Q4_K_M costs about 1.4 points of tool-calling reliability on an 8B model (0.933 to 0.919 in Docker's empirical test, as reported by RunAIHome 2026) and under 1.5 percent on task benchmarks; stepping up to Q6 or Q8 is the fix if schema violations appear, and gpt-oss is natively MXFP4 and loses nothing further (OpenAI 2025; RunAIHome 2026).\n\n## 5. What I Have Measured Myself\n\nThree artifacts from my own practice bear on the bet, and each one, re-read against the published evidence, turns out to have been telling me the answer already.\n\nThe first is a hierarchical multi-agent system, 18 role-specialized agents under a single orchestrator, which cut per-module delivery time by roughly 55 percent across more than 48 measured sessions. I had credited the harness. The literature suggests a correction: the system worked because deterministic orchestration, not any single model's memory, carried the long horizon. Each agent ran a short, decomposed, single-purpose loop, which is precisely the shape Section 3.2 says weak-model loops must take.\n\nThe second is the prompt-writer pattern inside that system: a Haiku-grade model, handed the narrow task of compressing context for sub-agents, cut sub-agent token consumption by 40 to 60 percent and dropped retries from 1.5 to 0.6 per task. This is Section 2 in miniature. A down-tier model plus a narrowed, verifiable task is reliable. I was never getting frontier judgment out of the Haiku-grade model; I was getting frontier-adjacent throughput on a task shaped to fit it.\n\nThe third is the pairing the bet was born from. My secondBrain, the vectorless knowledge base with deterministic two-step retrieval, is the knowledge component that Section 2.1 says transfers best. And Project Hydra, the sovereign personal AI system whose independence motivation pulls toward local models, already carries a locked decision that points the other way for now: cloud API for the reasoning core, local sovereignty as the direction rather than the current state. That decision, made on engineering grounds before I ran this synthesis, is the same conclusion the published evidence reaches. The two projects are separate artifacts; the lesson they converge on is one.\n\n## 6. The Engineering Answer: Tiered Routing\n\n### 6.1 Concede the Bet, Keep the Components\n\nThe original bet fails as stated. No harness makes a MacBook-resident model think like a frontier model on multi-turn, long-horizon, novel-reasoning work. But three of its five components survive intact (knowledge, format, verifiable tasks), and FrugalGPT's cascade result shows how to build with exactly those survivors: route by task shape, and keep the frontier model as the backstop rather than pretending it away. Escalation is a feature of the architecture, not an apology for it.\n\n### 6.2 Seven Design Rules\n\nRanked by impact per unit of effort, the rules the evidence supports:\n\n**Build retrieval first; it is the harness component that transfers best.** Knowledge substitution is the one near-full closure in the matrix (Izacard et al. 2022), and a deterministic knowledge base degrades gracefully as models shrink. Judgment does not.\n\n**Route by task shape, with frontier escalation as a designed feature.** Mechanical bulk goes local or down-tier: retrieval, formatting, drafting, summarization, test loops, anything with a cheap verifier. Judgment-grade work (long-horizon agentic tasks, consolidation-quality writing, decisions with cascading consequences) goes frontier. The cascade keeps the top tier as backstop (Chen et al. 2023).\n\n**Build verifiers before samplers.** Repeated sampling pays only where an automatic check exists (Brown et al. 2024; Snell et al. 2024), so build the verifier suite (schema checks, link validators, test gates, lint rules) first. Where no oracle exists, do not expect best-of-N to rescue a weak model, do not substitute an LLM judge at the same weak tier (verification quality is itself model-bound), and per Huang et al. (2024), do not let the model grade its own work.\n\n**Respect the multi-turn wall.** Below the roughly 30B-active class, multi-turn tool coherence collapses (TinyLLM Survey 2025). Keep local agent loops short, decomposed, and single-purpose, and let deterministic orchestration (scripts, staged workflows, hierarchical dispatch) carry the long horizon instead of the model's own memory. My 18-agent system is operational evidence, though not a controlled study, that this holds at production intensity.\n\n**Add harness structure only where a measured failure justifies it.** Harness-knob effects are non-monotonic (21 of 36 HAL runs got worse with more reasoning effort), and a 100-line loop beat heavyweight harnesses at the frontier (HAL 2025; SWE-agent Team 2025). The mega-harness is an anti-pattern.\n\n**Treat model swappability as a real feature with a real maintenance bill.** Every swap re-tunes prompt formats, tool-call dialects, context budgets, and retry behavior. I underpriced that bill when I made the bet.\n\n**Make the routing boundary a config value, not an architectural constant.** The laptop ceiling is Sonnet-class today and roughly 12 months behind frontier, with METR horizons doubling every 105 to 210 days. The boundary moves about twice a year; the architecture should expect it to.\n\n### 6.3 The One-Model-Generation Rule\n\nThe rule of thumb the numbers support: from a naive baseline, a good harness is worth roughly one model generation, a 2 to 6x multiplier on the tasks where scaffolding bites. Once a competent harness is in place, swapping the model dominates every other intervention. Spend the first unit of effort on the harness. Spend every unit after that on model access.\n\n## 7. Evidence Quality Caveats\n\nThe 2024 to 2025 findings this paper leans on (SWE-agent, Agentless via the SWE-bench Verified report, Snell et al., Large Language Monkeys, Atlas, the self-correction result, METR) are primary-source and load-bearing. The mid-2026 figures (frontier SWE-bench Verified scores up to 88.7 percent, GLM-5 at 77.8 percent, the per-model tokens-per-second numbers) come from third-party aggregators, community benchmarks, and practitioner write-ups rather than vendor model cards (the gpt-oss card, OpenAI 2025, is the exception), and they vary with context length, quantization, and runtime. I treat them as order-of-magnitude anchors, not precision claims. Nothing in these caveats changes the shape of the conclusion; it widens the error bars on individual cells.\n\nI also claim no new experiments. The practitioner numbers in Section 5 are operational measurements from my own systems, reported as practice evidence, not as controlled studies.\n\n## 8. Conclusion\n\nI bet that a strong harness around a swappable weak model would produce frontier behavior, and the published evidence forced me to grade that bet honestly: partially right, and wrong exactly where it mattered most. Retrieval substitutes for knowledge almost fully. Schemas substitute for format reliability. Verification plus sampling substitutes for capability wherever an oracle exists, to the tune of a 15x swing on SWE-bench with weights held constant. But multi-turn coherence, long-horizon autonomy, and novel hard reasoning are model-bound, with same-harness spreads of 1.5 to 4x on agentic coding and 50 to 100x on autonomous time horizons, and no scaffold in the literature closes them.\n\nThe architecture that survives is not \"harness until frontier.\" It is tiered routing: a deterministic knowledge base and a verifier suite as the permanent substrate, local or down-tier models for the bulk work those components make reliable, and frontier escalation for judgment, held behind a boundary that lives in config because the trailing edge moves twice a year. The harness is not the model. It is worth roughly one model generation, which is a great deal, and it is not more than that. That is not a defeat for harness engineering. It is the job description: build the harness that makes every tier cheap where it can be cheap, and route the rest up.\n\n## References\n\n*Llama 4 vs Qwen 3.5 vs Gemma 3: Which open model should you deploy?*https://ai.rs/ai-developer/llama-4-vs-qwen-3-5-vs-gemma-3-compared\n\n*Berkeley function-calling leaderboard.*https://gorilla.cs.berkeley.edu/leaderboard.html\n\n*Large language monkeys: Scaling inference compute with repeated sampling.*arXiv:2407.21787. https://arxiv.org/abs/2407.21787\n\n*FrugalGPT: How to use large language models while reducing cost and improving performance.*arXiv:2305.05176. https://arxiv.org/abs/2305.05176\n\n*SWE-bench scores: How leading models and harnesses compare.*https://www.codeant.ai/blogs/swe-bench-scores\n\n*Holistic Agent Leaderboard: Standardized evaluation of agent harnesses.*arXiv:2510.11977. https://arxiv.org/abs/2510.11977\n\n*Practitioner thread on running gpt-oss-120b locally (under \"Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs\").*https://news.ycombinator.com/item?id=44822195\n\n*Large language models cannot self-correct reasoning yet.*ICLR 2024. arXiv:2310.01798. https://arxiv.org/abs/2310.01798\n\n*Atlas: Few-shot learning with retrieval augmented language models.*arXiv:2208.03299. https://arxiv.org/abs/2208.03299\n\n*Pairing Claude Code with local models.*https://www.kdnuggets.com/pairing-claude-code-with-local-models\n\n*Claude Code with local backends: Setup notes and the KV-cache invalidation fix.*https://gist.github.com/kibotu/a009f00414b7c10fb1c74e603d7838c0\n\n*LLM benchmarks: Aggregated model scores.*https://llmcheck.net/benchmarks\n\n*Open and closed model leaderboard.*https://www.marc0.dev/en/leaderboard\n\n*Qwen 3 on M4 Max: Throughput benchmark.*https://markaicode.com/benchmarks/hugging-face-qwen-3-m4-max-throughput-benchmark/\n\n*Measuring AI ability to complete long tasks.*https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/\n\n*Time horizons.*https://metr.org/time-horizons/\n\n*Introducing SWE-bench Verified.*https://openai.com/index/introducing-swe-bench-verified/\n\n*gpt-oss-120b and gpt-oss-20b model card.*arXiv:2508.10925. https://arxiv.org/html/2508.10925v1\n\n*Quantization quality loss in 2026: Q4, Q5, Q6, Q8 measured.*https://runaihome.com/blog/quantization-q4-q5-q6-q8-quality-loss-2026/\n\n*Qwen3-Coder-Next: The complete 2026 guide to running powerful AI coding agents locally.*https://dev.to/sienna/qwen3-coder-next-the-complete-2026-guide-to-running-powerful-ai-coding-agents-locally-1k95\n\n*tau-bench: A benchmark for tool-agent-user interaction in real-world domains.*https://github.com/sierra-research/tau-bench\n\n*Scaling LLM test-time compute optimally can be more effective than scaling model parameters.*arXiv:2408.03314. https://arxiv.org/abs/2408.03314\n\n*mini-swe-agent: A 100-line agent that scores over 74 percent on SWE-bench Verified.*https://github.com/SWE-agent/mini-swe-agent\n\n*Why gpt-oss-120b feels slow on a Laptop M4 Max 128GB.*https://developer.tenten.co/why-gpt-oss120b-feels-slow-on-a-macbook-pro-m4-max-128gb\n\n*A survey of small language models as tool-calling agents.*arXiv:2511.22138. https://arxiv.org/pdf/2511.22138\n\n*M4 Max 128GB: What it runs.*https://willitrunai.com/macs/m4-max-128gb (companion guide: https://willitrunai.com/blog/qwen-3-5-mlx-apple-silicon-guide)\n\n*The biggest local LLM on your machine is useless if it can't call a single tool, no matter how many parameters it has.*https://www.xda-developers.com/biggest-local-llm-machine-useless-cant-call-single-tool-how-many-parameters/\n\n*SWE-agent: Agent-computer interfaces enable automated software engineering.*arXiv:2405.15793. https://arxiv.org/abs/2405.15793", "url": "https://wpnews.pro/news/the-harness-is-not-the-model-how-far-scaffolding-takes-a-weak-llm", "canonical_source": "https://manazir.dev/work/the-harness-is-not-the-model", "published_at": "2026-07-13 07:18:40+00:00", "updated_at": "2026-07-13 07:35:23.532750+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-research", "ai-infrastructure", "developer-tools"], "entities": ["METR", "SWE-bench", "NaturalQuestions"], "alternates": {"html": "https://wpnews.pro/news/the-harness-is-not-the-model-how-far-scaffolding-takes-a-weak-llm", "markdown": "https://wpnews.pro/news/the-harness-is-not-the-model-how-far-scaffolding-takes-a-weak-llm.md", "text": "https://wpnews.pro/news/the-harness-is-not-the-model-how-far-scaffolding-takes-a-weak-llm.txt", "jsonld": "https://wpnews.pro/news/the-harness-is-not-the-model-how-far-scaffolding-takes-a-weak-llm.jsonld"}}