The harness matters more than the model weights now

A production engineer reports that swapping Claude 3 Opus for Haiku in a customer-support agent cost only 3% resolution rate after harness improvements, while a 3B local model with a TypeScript orchestration layer achieved 78% task completion on a 50-scenario eval suite versus GPT-4o's 71% with a naive prompt chain, arguing that the control loop around the LLM, not the model weights, determines performance.

The harness matters more than the model weights now This aligns with what I've seen in production. Six months ago I watched a team swap Claude /en/tags/claude/ 3 Opus for Haiku in a customer-support agent and lose maybe 3% resolution rate — after they rewrote the prompt chain, added structured output validation, and built a proper fallback ladder. The model downgrade was noise. The harness rewrite was signal. What the Nvidia paper makes explicit: you can generate synthetic trajectories that cover the failure modes your real users will hit, fine-tune a small open model on those, and deploy something that stays on rails. The "agent" isn't the LLM. The agent is the control loop around the LLM. That loop — planning, tool selection, argument validation, error recovery, context compression — is software engineering, not prompt engineering. Practical takeaway for anyone building a hands-on guide for their next AI workflow: stop benchmarking base models on MMLU. Start benchmarking your harness on realistic task distributions. Build an eval set that mirrors actual user intent — ambiguous instructions, missing parameters, tool timeouts, contradictory context. Then iterate the harness until pass rate climbs. The model is a commodity. The harness is your moat. I'm currently stress-testing this with a 3B model running locally via Ollama, wrapped in a thin TypeScript orchestration layer. Early numbers: 78% task completion on a 50-scenario eval suite that includes parallel tool calls and mid-stream corrections. Same suite, GPT-4o with a naive prompt chain: 71%. The difference isn't intelligence. It's discipline encoded in the harness. Is the AI rally a genuine productivity boom or a 3h ago /en/news/7282/ Nvidia's latest demo proves the inference stack matters more 12h ago /en/news/7216/ Nvidia AVO hits 100% on ARC-AGI-3 and the benchmark might be 20h ago /en/news/7169/ Nvidia's compute asset class push hits $500 billion — here's why 1d ago /en/news/7102/ Cerebras WSE-3 smokes H100 on Llama 3 70B inference at a 2d ago /en/news/6962/ Nvidia's buyback binge looks smart until you check the 2d ago /en/news/6956/ Next Michael Jackson vinyl exhibition at sspai Guangzhou store shows → /en/news/7290/