{"slug": "debugging-on-weak-models-reveals-more-bugs", "title": "Debugging on weak models reveals more bugs", "summary": "Archestra, an AI agent platform, deliberately runs its Archestra Chat interface on weaker, cheaper models to expose product bugs that strong models hide, and has built a nightly 26-task benchmark based on real customer workflows to validate fixes. The approach has found defects in file handling, sandbox tooling, provider schemas, and the agent runtime, improving reliability for all models. The benchmark, which checks answers against hidden ground truth using deterministic code, answers how far down the price ladder the assistant remains useful.", "body_md": "2026-07-20\n\n# We Debug Our AI Harness on Weak Models on Purpose\n\nHow we use weak, cheap models to find bugs strong models hide - and what a nightly 26-task benchmark says about which ones you can trust with real work.\n\nWritten by\n\nArseny Kravchenko\n\nStrong models are excellent at hiding product bugs. Give one a broken tool call or a confusing error, and it may still finish the task.\n\nWeaker models are less forgiving. When the tools, prompts, or runtime are wrong, they fail quickly, which makes them useful for debugging. I think of it as the difference between testing an app on a maxed-out MacBook Pro and testing it on an old ThinkPad.\n\nWe run\n\n[Archestra Chat](https://archestra.ai/docs/platform-chat?q=chat)(our built-in chat interface for working with agents and MCP tools) on weaker models on purpose, record the full trajectory, and investigate the failures they expose. This approach has found defects in file handling,[sandbox tooling](https://archestra.ai/docs/platform-code-sandbox?q=sandbox),[provider schemas](https://archestra.ai/docs/platform-adding-llm-providers?q=provider%20schema), and our[agent runtime](https://archestra.ai/docs/platform-agents). Fixing those bugs helps every model - not just the weaker ones. Strong models no longer need to spend extra tokens and retries working around our mistakes. As a result our harness works reliably even with weaker models (not only models capable of compensating for its flaws).The idea came from a customer who asked me a simple question: would Archestra Chat work with cheaper models? I couldn't answer. At the time, we were testing changes one at a time, adjusting tools and prompts, and checking whether the final answer looked right. Sometimes it worked. Sometimes it didn't. We had no repeatable way to measure it. I'm an ML person at heart, and when a system behaves like that, the first thing I build is a validation loop.\n\nThat loop became an\n\n[end-to-end benchmark](#so-which-models-held-up-open-weight-vs-frontier-results)based on 26 real customer workflows. It starts the actual product, runs each task, and checks the answer against hidden ground truth. Every night, it repeats the same tasks across about ten models.**Important to note:** this is not an academic benchmark attempt. But it's an advanced integration test that answers two questions: where does our product fail, and how far down the price ladder can we go before the assistant stops doing useful work?How we benchmark AI agents on real work, not party tricks\n\nI approached this the way you'd approach any serious evaluation: the test has to be something the system can't cheat, and it has to look like reality. That leads to two design pillars.\n\nFirst, the assistant submits its final answer through a dedicated channel. This channel checks the format, not whether the answer is correct. If the task expects a number, it rejects text. If it expects three candidates, it rejects two.\n\nThe answer is checked for correctness only after submission, using deterministic code, not another LLM. The grader and expected answer are hidden from the agent during the run. If the agent could read the answer key, it could game the test, and then the score would be a lie. It can't, so the score means something.\n\nThe boundary that makes the score mean something:\n\nLoading diagram...\n\n`submit_result`\n\nnever tells the agent whether it was right - only whether the JSON parsed against the task's schema. The expected ground truth lives entirely on the right of that line and is never staged into the sandbox, so the agent has nothing to read off.Second, we don't record just pass or fail. We capture the assistant's entire visible trajectory: every message, tool call, file it touched, and artifact it produced. That turns “it failed” into “here's exactly where it failed.” It's the difference between a number that shames you and a report you can actually use. More on that shortly, because this became the most useful thing we built.\n\nThe tasks come from reality, not our imagination. Each one is based on something we actually saw: a real customer workflow or a real incident, lightly anonymized. We have a small set today, and it grows whenever reality hands us a new headache. Once a task goes in, it stays in, a permanent tripwire for that failure mode.\n\nA few examples, to make this concrete:\n\n**Document workflows.** Approve invoices and shortlist CVs from mixed file formats while rejecting[hidden prompt injections](https://archestra.ai/blog/prompt-injection-examples)and inconsistent data.**Incident triage.** Find the cause of an outage in a zip of unsorted logs containing plausible red herrings.**Memory across conversations.** Create a file in one conversation, then retrieve and use it in a new one - this exercises the[persistent files](https://archestra.ai/blog/new-projects-filesystem)we shipped in July.**Live facts.** Fetch a repository's current star count, an asset price at a specified time, or the latest package version.\n\nWe've tightened them to match how a real employee actually talks: “Here are some invoices. Do the right thing.” That harder version is the one worth measuring.\n\nWe also keep a hard boundary around what belongs in the suite. Tasks should be difficult, but solvable with the tools Archestra Chat actually has: file access, web access where needed, the\n\n[sandbox and skills](https://archestra.ai/blog/new-skills-sandboxes), and the customer's connected systems. We're not measuring whether a model can do frontier physics, and we're not testing features we haven't built yet. The useful failure mode is: “The product should have been able to do this, but didn't.”One implementation detail matters a lot: this is not a mock benchmark. It boots the product. The benchmark runs Chat against a real Archestra backend in headless mode. For each environment, it starts a fresh backend on a new port, migrates a fresh database, seeds the providers, skills,\n\n[MCP servers](https://archestra.ai/mcp-catalog), and agent surface, drives chat sessions through the product, grades the answers out of band, and tears the instance down.The harness is public: it lives in the\n\n[, alongside the benchmark, task suite, and environment definitions.](https://github.com/archestra-ai/archestra/tree/main/ai-labs)`ai-labs`\n\nworkspace of our monorepoIt runs every night in CI against that day's deployed build, across a roster of about ten models. The nightly series is where the numbers later in this post come from.\n\nA single run, end to end:\n\nLoading diagram...\n\nEvery box above serves the benchmark's one job: run each task-model combination, produce a graded result and a recorded trajectory, then tear everything down. Nothing here explains why a run failed. That comes later, in a separate tool.\n\nAnyone can ship a demo that runs on a maxed-out MacBook\n\nHere's the part that surprised me.\n\nThink about software performance for a second. Any developer can write an app that feels smooth on the latest MacBook Pro - the machine is so fast it hides your sins. Lazy code, wasteful operations, sloppy logic: none of it shows, because the hardware brute-forces past it. Then you try to run that same app on an old IBM ThinkPad with the red TrackPoint and a spinning hard drive, and it falls apart instantly. Every shortcut you took is suddenly visible.\n\nPremium AI models are the maxed-out MacBooks. They can bulldoze through bad plumbing - a malformed tool call, a confusing error message, a missing fallback - and make the product look healthier than it is. The model works around the bug, your benchmark comes back green, and you learn nothing.\n\nSo we did our debugging on the metaphorical old ThinkPad. We deliberately ran the benchmark against weak, cheap models, because they don't self-heal. When something in our plumbing is even slightly wrong, a weak model trips over it immediately and loudly - which means it shows us exactly what to fix. The weak models are a better smoke alarm precisely because they're worse at compensating for our mistakes.\n\nThis isn't just a clever debugging trick. Software that survives the old ThinkPad is the software that runs well everywhere. A model that compensates for our bugs is still paying for them in retries, tokens, latency, and fragility. Hardening Chat against the weakest models is how we stop taxing the strongest ones. The test method and the business goal turn out to be the same thing.\n\nYou can see that tax in the results table below: two models with similar performance can differ in price by two orders of magnitude.\n\nThe real bugs weak models found for us\n\nIf I'm honest, the benchmark's biggest contribution so far hasn't been the scoreboard. It's been a bug-finding machine, and most of those bugs were ours, not the models'.\n\nI read the first recordings by hand. After around ten nearly identical failures, I gave up and spent a week building a separate analyzer on top of them. The benchmark grades and records each run. The analyzer then reads those recordings, summarizes what appears to have gone wrong, groups similar failures, and points engineers toward the relevant parts of the product.\n\nThe important word is\n\n*appears*. The analyzer produces hypotheses, not truth, so a human still reviews the trajectory, traces the source code, confirms the failure, and decides what to fix. That human step is what turns “the benchmark failed” into a real product improvement instead of an automated blame machine.It's a map-reduce process over the recorded trajectories. It starts where the previous diagram ended, using the same trajectory files. It only accesses the live product as read-only source code to cross-check its findings against.\n\nLoading diagram...\n\nThe hand-off between the two tools is a single artifact: the recorded trajectory. The benchmark never tries to identify the root cause, and the analyzer never runs the product. Keeping them separate lets a human use the analyzer's leads without trusting them blindly.\n\nHere's a representative sample of issues we found and fixed through that loop in a single week, limited to defects you can understand without knowing our prompt stack:\n\n| Area | What broke | What changed |\n|---|---|---|\n| File handling | Binary uploads could crash the LLM request even though the files were already available in the sandbox. | Keep those files as sandbox references instead of trying to inline unsupported MIME types.\n|\n\n[#5697](https://github.com/archestra-ai/archestra/pull/5697)`zip`\n\nor `unzip`\n\n.`zip`\n\nand `unzip`\n\nso Chat can inspect archives directly. [#5707](https://github.com/archestra-ai/archestra/pull/5707)[#5756](https://github.com/archestra-ai/archestra/pull/5756),[#5786](https://github.com/archestra-ai/archestra/pull/5786)`finish_reason`\n\nstring our API schema did not recognize, so trajectory playback failed.[#5780](https://github.com/archestra-ai/archestra/pull/5780)[#5749](https://github.com/archestra-ai/archestra/pull/5749),[#5787](https://github.com/archestra-ai/archestra/pull/5787)[#5797](https://github.com/archestra-ai/archestra/pull/5797),[#5801](https://github.com/archestra-ai/archestra/pull/5801)These fixes came from running hundreds of rollouts through the same product surface, not from engineers manually clicking around Chat. Unit and integration tests remain necessary, but they rarely exercise files, tools, sandboxes, provider quirks, and persistence in one path, and do not bring enough variance to cover a non-deterministic system.\n\nSo which models held up? Open-weight vs. frontier results\n\nThe question I get when talking to our customers is: do we really have to budget premium models for every seat?\n\nWhen export controls\n\n[forced Anthropic to suspend access to Claude Fable 5 in June](https://www.anthropic.com/news/redeploying-fable-5), we bet every executive who has ever rolled out AI had the same thought: what gets cut off next, and for whom? Access came back three weeks later, but that box doesn't close. You didn't have to use that model to see the risk - and now nobody gets to un-see it. Terms change, access gets revoked, regions get geofenced, prices move.[Open-weight models](/blog/os-models-for-enterprise-agents)- models you can run yourself instead of depending on one vendor's permission slip - are the obvious hedge.But a hedge you can't trust isn't a hedge. So we test which cheaper, self-hosted models can handle real employee tasks. That's the benchmark's second axis: we run the same tasks on models ranging from premium APIs to small open-weight models.\n\nThe results match what we expected - and a benchmark that agrees with reality where you can already check it is one you can believe where you can't.\n\nEvery model below ran the same 26 tasks against the latest daily build. Small changes between builds can add some variance; we're not aiming for full determinism here.\n\nFor models on our daily roster, we report the average pass rate across seven consecutive daily runs on the same frozen task set, along with the typical variation between runs. Frontier models were run once on the same day and tasks, and are marked accordingly.\n\n| Model | Open weights | Pass rate | Day-to-day swing, pp1 | Suite cost2 |\n|---|---|---|---|---|\n| Claude Sonnet 5 | 100% | single run | $19.95 | |\n| Claude Opus 4.8 | 96% | single run | $27.60 | |\n| GPT-5.6 (sol) | 96% | single run | $7.50 | |\n| GPT-5.6 (terra) | 96% | single run | $3.93 | |\n| Qwen3.7 Plus | ● | 94% | ± 2 | $0.57 |\n| Sakana Fugu-Ultra | 92% | single run | $27.69 | |\n| GLM-5.2 | ● | 91% | ± 7 | $1.23 |\n| DeepSeek V4 Flash | ● | 88% | ± 7 | $0.34 |\nClaude Fable 53 | 88% | single run | $46.17 | |\n| Xiaomi MiMo v2.5 | ● | 86% | ± 4 | $0.27 |\n| Qwen3.6 27B | ● | 85% | ± 4 | $1.19 |\n| GPT-5.6 (luna) | 81% | single run | $1.62 | |\n| Qwen3.6 35B-A3B | ● | 71% | ± 10 | $0.82 |\nClaude Haiku 4.54 | 68% | ± 6 | $7.82 | |\n| Gemma 4 31B | ● | 45% | ± 8 | $0.57 |\n| GPT-5.4 nano | 12% | ± 6 | $0.37 | |\n| GPT-5.4 mini | 9% | ± 3 | $0.90 |\n\n**Notes**\n\n1Standard deviation across seven daily runs from July 3-9. On a 26-task suite, each task is worth about four percentage points. So ±7 means a typical run varies by roughly two tasks from the average. “Single run” means one measurement, not a stable average.\n\n2Cost of one complete 26-task run through\n\n[OpenRouter](/blog/openrouter-with-archestra).\n\n3Clean rerun. An earlier attempt that day failed at boot because of a harness error, so we excluded it.\n\n4Average across four daily runs from July 3-6, before the model left the daily roster. The cost is also averaged across those four runs.\n\nFocus on the ranges, not the decimals. With only 26 tasks, one result can move the score by about four points. Small differences don't mean much.\n\nThree important findings:\n\n**The strongest open models overlap with the frontier tier.** The cheapest model in that range costs $0.34 per run, compared with $20-28 for frontier models.**Below roughly 70%, the wheels come off entirely.****Price and capability don't follow the same order.** One brand-name mid-tier model costs $7.82 per run but scores below five open models that cost between $0.27 and $1.23.\n\nOne outlier is worth calling out. Qwen3.6 35B-A3B scored between 58% and 88% across the week, despite running the same tasks each day. A single-run benchmark would have hidden that instability.\n\nThe takeaway for decision-makers: stop asking, “Which premium model should we buy for everyone?” Ask, “What's the cheapest model that can still handle our actual work?”\n\nIn our benchmark today, an open-weight model passes roughly 90% of the tasks for well under $1 per run. Frontier models cost $20-30 per run for the remaining few percentage points.\n\nSo no, you may not need a premium model for every seat. Our numbers are ours - your tasks will rank models differently. But the benchmark is public, so\n\n[steal the good bits](https://github.com/archestra-ai/archestra/tree/main/ai-labs).Where this is going\n\nTo be clear, this is still early. The suite has 26 tasks - enough to see broad performance ranges, but small enough that one task moves the score by four points. We'll keep adding tasks based on real customer problems.\n\nThe benchmark already runs in CI. Next, we want to make it harder to ignore: broader coverage, clear release gates, and better reports when something regresses. The harder problem is orchestration: can we help smaller models get closer to the larger ones without hiding problems behind retries and extra tokens? That needs proper research. We'll write about it when we have results.\n\nBefore this, we couldn't answer a basic question: does Chat actually work across different models? Now we have repeatable runs, failures broken down by model and task, and trajectories that lead us back to real bugs. That's what we need to make Chat reliable for someone who doesn't know - or care - which model is underneath.", "url": "https://wpnews.pro/news/debugging-on-weak-models-reveals-more-bugs", "canonical_source": "https://archestra.ai/blog/we-debug-our-ai-harness-on-weak-models-on-purpose", "published_at": "2026-08-03 13:03:17+00:00", "updated_at": "2026-08-03 13:22:44.022831+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "ai-tools", "ai-infrastructure"], "entities": ["Archestra", "Archestra Chat", "Arseny Kravchenko"], "alternates": {"html": "https://wpnews.pro/news/debugging-on-weak-models-reveals-more-bugs", "markdown": "https://wpnews.pro/news/debugging-on-weak-models-reveals-more-bugs.md", "text": "https://wpnews.pro/news/debugging-on-weak-models-reveals-more-bugs.txt", "jsonld": "https://wpnews.pro/news/debugging-on-weak-models-reveals-more-bugs.jsonld"}}