# Cutting our agent's harness cost 10x without breaking the product > Source: > Published: 2026-09-18 15:08:21+00:00 # Cutting Our Agent's Harness Cost 10x Without Breaking the Product Like many AI startups, we were watching our inference bill grow with usage. Our agent at Notch makes video ads, so we expected video generation to be expensive. What surprised us was that roughly **40% of our AI spend** went to the Sonnet-powered harness coordinating the work. We thought we could do better. After a two-week sprint, we switched to GPT-5.6 Luna. Median harness cost in sessions that produced a video fell from **$4.44 to $0.50**—nearly 90% less. In this post we talk through how we tested the alternatives, why Luna won, and what the benchmarks didn’t tell us. We also found that models have personalities: two can pass the same evals and still turn the same brief into very different ads. ## **1. Your SDK doesn’t have to lock you into its model** We chose the [Claude Agent SDK](https://code.claude.com/docs/en/agent-sdk/overview) in February 2026. Claude was the strongest model we’d tried for multi-step tool use, and the SDK gave us Claude Code’s agent loop, context compaction and resumable sessions. It was the fastest way to start with a mature harness. By the time we wanted to try cheaper models, we had built over 100 tools around the SDK—from casting avatars to editing timelines—plus skills and specialized subagents. We expected switching providers to mean replacing the harness. But luckily we found out the SDK can call another server through `ANTHROPIC_BASE_URL`, provided it speaks Anthropic’s Messages API. Some providers supported that API directly. For the others, we used [LiteLLM Proxy](https://docs.litellm.ai/docs/anthropic_unified#litellm-proxy-server), an open-source LLM gateway we ran alongside the agent. It translated requests and responses between the SDK’s Messages API and the provider’s API. *The SDK continues to run the loop. LiteLLM translates the conversation with the model.* **Lesson:** You may be able to change models at the API boundary while keeping the harness you’ve already built. ## **2. Popular benchmarks can be misleading** We used SWE-bench Pro as a rough proxy for planning and tool use. Our goal was to cut harness costs by 10×, so we looked for strong benchmark performance at much lower token prices. Four models looked promising: **GPT-5.6 Luna, MiniMax M3, Muse Spark 1.1 and DeepSeek V4 Pro.** We then tested them using our own evals and customer briefs. *These evaluations took place in July–August 2026. The observations below describe the model versions and provider endpoints we tested then; newer releases may behave differently.* *Circled: the four models we tested as replacements. Other models provide context. Historical output-token rates; log price axis. Vendor-reported scores use different harnesses. Sources:* *Sonnet 5**,* *Luna**,* *Muse**,* *MiniMax**,* *DeepSeek**,* *Opus 4.8**,* *Sonnet 4.6**,* *GPT-5.5**,* *Kimi**,* *Gemini**.* **MiniMax and Muse looked strong on SWE-bench, but struggled to make progress through our tools.** MiniMax got stuck in a validation-retry loop: in one street-interview task, it made roughly twenty attempts to save a scene plan, repeatedly submitting malformed arguments despite explicit error messages. It eventually blamed our serializer and gave up. Muse got stuck a step earlier: it kept searching for tools without invoking the ones needed to do the work. This surprised us: models that could fix difficult bugs were getting stuck saving a scene plan. The coding problems were hard, but the tools for doing the work could be as simple as [a command line](https://mini-swe-agent.com/latest/). Our scene tool was more like a form where each answer changed which fields had to be filled in next. Some models kept getting those fields wrong, even after being told what to fix. [Berkeley’s tool-use benchmark](https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html) tests this ability to follow tool requirements and recover from mistakes which surprisingly is orthogonal to performance on complex coding tasks. **DeepSeek impressed us with its speed and reasoning, but the version we tested couldn’t see.** At the time, the endpoint we used didn’t accept image inputs. That mattered when a customer supplied a reference video and asked our agent to create something inspired by it. We tried giving DeepSeek a tool that let it ask a vision model questions about an image. It could choose what to ask, so this went beyond feeding it a fixed caption. But it still struggled with these briefs. It was like a director who could ask an assistant about the footage, but never watch it themselves. **Luna was the best fit, but it struggled with nested union types in our tool schemas.** We replaced those unions with ordinary objects, reusing a fix we’d built while testing MiniMax. A simplified example: **Before: a union of object types.** **After: one object type.** Our code still validated every call against the original schema before saving the scene. **Lesson:** Use benchmarks to shortlist models, then test them on your own workload. Be prepared to adapt your harness: even a strong candidate may not work reliably out of the box. ## **3. Models have different personalities, even when they pass the evals** Once Luna passed our automated evals, we ran a final human review, comparing its finished ads with Sonnet’s on the same briefs. Both models could satisfy the brief, but the ads often looked quite different. The differences showed up in choices our harness left open: who the presenter should be, what they should wear and how the scene should look. Sonnet tended to spell out more of these details. Its avatar-generation prompts were about 19% longer in our production sample: a median of 623 characters versus Luna’s 522. *Median of each session’s median avatar-prompt length, excluding shared guardrail text. Observed production traffic; customer briefs were not matched.* Yet Luna’s shorter prompts sometimes produced more distinctive characters. The prompts help explain the result: each agent was choosing some details itself and leaving others to the image model. **Warehouse owner: “black t-shirt” versus “dark workwear”** Sonnet specified a “plain unbranded black t-shirt.” Luna asked for “practical dark workwear,” leaving the image model to choose the jacket, fabric and stitching. *Sonnet on the left; Luna on the right. “Dark workwear” left the cut, fabric and construction of the jacket open.* **Workshop presenter: specified features versus a broader character** Sonnet specified “short hair, light stubble” and an olive flannel over a grey tee. Luna asked for a “casual cap-and-workwear feel,” which emerged as a full beard, weathered cap and denim shirt. *Boot colors follow different product references: teal for Sonnet, dark green for Luna. These examples share opening requests and products, but reference assets and later edits differ; this compares character design, not a controlled rerun.* **A couple at home: a new outfit versus the reference’s style** Sonnet specified a cream cardigan, white tee and jeans. Luna kept the reference’s “glasses-and-headwrap styling” and asked for “casual home wardrobe,” leaving more of the look to the image model. *Same customer request and identical reference video. Sonnet on the left; Luna on the right.* Our download/publish rate stayed roughly unchanged after rollout. That mattered more to us than making Luna’s ads look exactly like Sonnet’s. **Lesson:** Passing the same evals doesn’t mean producing the same creative work. Review the finished outputs. If a particular style matters to your product, specify it in the harness. ## **4. Token price isn’t the cost of an outcome** The price table is where a migration starts. The full workflow is where the saving either survives or disappears. Our agent has a review-and-fix loop: inspect the output, identify a problem, revise the instructions and regenerate what needs fixing. That makes a bad creative decision a cost problem as well as a quality problem. Saving money on the LLM that writes a video prompt helps less if we have to pay for the footage twice. Here is a deliberately simple worked example. Suppose the harness costs $4 and media generation costs $6. A tenfold cheaper harness brings the total from $10 to $6.40. But if the cheaper model needs one extra $6 generation, the total becomes $12.40—more than where we started. The figures are illustrative; the review-and-fix loop is real. *Illustration, not measured model results. One extra generation can outweigh a large token saving.* One integration bug made this concrete. Optional tool fields were becoming required as our requests passed through the gateway into the Responses API. The model filled fields it should have omitted, causing invalid calls. We explicitly disabled [strict mode](https://developers.openai.com/api/docs/guides/function-calling#strict-mode) for that route and kept validation in our own tools. In one before-and-after planning-eval comparison, validation retries fell from 14 to 4. The model and its token price had not changed. We encountered a less obvious version of this with Muse. It struggled with deferred tool search, spending calls looking for tools instead of doing the work. Loading the full catalog up front helped, but put more tool definitions into the context. The token price stayed the same; the number of tokens we needed changed. Caching can swing the calculation the other way. Agents resend long prefixes on successive turns: instructions, tool definitions and conversation history. DeepSeek caches shared prefixes automatically. At the rates in our migration’s price registry, its cached input was 120 times cheaper than uncached input—$0.003625 versus $0.435 per million tokens. Those are historical rates, not a current quote. The durable lesson is to measure the cache hits your real trajectories get. [DeepSeek caching](https://api-docs.deepseek.com/guides/kv_cache/) · [Current pricing](https://api-docs.deepseek.com/quick_start/pricing/) We also learned to distrust a dollar figure until we knew how it was calculated. The SDK’s price table made one Luna turn look like it cost $3.18. Applying our own model registry to its recorded usage gave about $0.13. Separately, the gateway was dropping cache-read usage from its reports, so cached context appeared to be full-price input. Either bug could have distorted which model looked economical. Our observed median harness cost fell from $4.44 to $0.50 in completed-video sessions. Holding the token mix constant and changing only the base rates gave a 9.86× difference. Those answer different questions: what the sampled sessions cost, and how much cheaper the same usage would be. To measure the full cost of an accepted ad, include generation, review, retries and unsuccessful attempts too. **Lesson:** Compare models by the full cost of producing a usable ad, including retries and regeneration. ## **Making the next switch easier** We made the switch through three gates: **Gate 1: Can it operate the harness?** Our automated evals checked whether the model could use our tools, recover from errors and complete the task. **Gate 2: Is the ad still good?** Human review checked the finished ads, including the creative choices our automated evals didn’t judge. **Gate 3: Does it survive production?** A staged rollout let us compare the full cost and customer outcomes with Sonnet. We now send 95% of traffic to Luna and keep 5% on Sonnet as a long-term holdout. The download/publish rate has stayed roughly unchanged. We’re continuing to track production costs and downstream effects on subscriptions, which can take longer to show up. The options have already changed since we ran these tests. [DeepSeek has released V4.1 Flash with native image input](https://www.deepseek.com/en/news/deepseek-v4-1-flash/), giving us a reason to revisit a family we previously ruled out. We can now put that candidate through the same gates, using the harness and evals we’ve already built. ## Your competitors are hiring. So should you. Book a demo to discuss your team’s workflow and enterprise options.