We ran Terminal-Bench 4.0 11% cheaper and 12% faster than the frontier labs

All posts Benchmarking Omnara with Terminal-Bench 4.0

On this page #

Building a reliable, performant harness is difficult, so difficult that some say only the labs can do it well. Fortunately, this isn’t true: on Terminal-Bench 4.0 our Omnara harness matched the pass rates of Claude Code and Codex while costing less and for some models completing faster!

Omnara is an open-source, model-agnostic platform built for deploying and running agents. As a part of that we've built our own harness, primarily for infrastructure reasons rather than LLM performance reasons. Omnara's agent loop functions directly on our cloud and interfaces with a target machine, rather than running locally inside an environment. This allows our agent to be durable, resumable, and horizontally scalable.

To achieve these cost and time savings, we’ve tuned our open-source harness and tool architecture for token efficiency, without sacrificing on performance. We’ve made our results and benchmark harness fully accessible and want to show how we achieved this with our improved tool sets.

Running the benchmark #

Omnara works with any model, so we decided to test the model against its lab’s own harness. We ran Terminal-Bench 4.0 with two frontier models, Claude Opus 5.5 and GPT 6.1 Sol, both at max reasoning effort. We compared the results against its lab’s own harness: Claude Code for Opus and Codex for Sol.

Claude Code and Codex are CLIs that run inside the task’s sandbox. Omnara’s agent loop runs on our cloud instead, and reaches a machine through omnarad, a small daemon. For each task, Harbor, Terminal-Bench’s task runner:

• Starts a Modal sandbox, with omnarad preinstalled¹

• Launches an Omnara agent with that sandbox attached as its machine

• Evaluates the results in the sandbox once the Omnara agent has completed its turn² Attaching a sandbox to an agent config gives the agent Omnara’s standard machine toolset, described in the tool architecture section below.³ Omnara is a general agent, so it doesn’t come with a default system prompt. Instead, we adapted the Claude Code and Codex system prompts to Omnara’s feature set for our runs. Both models ran at max reasoning effort with a 128k output cap, matching their native harnesses. We disabled web search and fetch for our Omnara runs.

Cost comparisons #

In the Omnara harness, Claude Opus 5.5 (max) has a 6% increase in pass rate over Claude Code⁴. It gives that performance faster, at a marginally lower cost. It cost around 5% less and takes 12% less time to complete its tasks. In certain extreme cases, like in the task embedding-drift-monitor, both Omnara and Claude Code solve the task consistently, but Omnara is significantly more efficient. For Claude Code, it’s a nearly 1 hour task costing $8.54 on average, but for Omnara, it’s a task that spends half the time and half the money, mainly due to fewer LLM API calls (44 vs 63).

With GPT 6.1 Sol, the Omnara harness performs within the margin of error of the Codex harness.⁵ But, it has an even wider efficiency gulf to Codex. The cost per task of the Omnara harness ends up 11% less than in Codex, with 35% fewer tokens (5.03M → 3.26M tokens). In fp8-rmsnorm-gemm, both Omnara and Codex solve the task consistently, but Omnara is again more efficient. For Codex, it’s a task costing $2.01 on average, but for Omnara, it costs $1.29, about two-thirds as much. The primary reason is Omnara executes 33% fewer LLM API calls and uses 55% fewer tokens.

Why is Omnara cheaper? #

If the labs have so much effort invested in their harnesses, how is our harness able to perform more efficiently? The trick that Omnara uses is that it defers most context, especially when executing processes. Omnara exposes a slightly different toolset from most other harnesses. Rather than a single blocking exec tool, Omnara exposes a run_command tool, which does not block the agent loop for long commands. Instead, it returns a process_id, which the agent can pass into other process management tools. Then, the agent can use tools like read_process, write_process and watch_process to interface with the process later.6

This tool architecture results in a few benefits:

1. Waiting without polling: Most Terminal-Bench tasks are designed to last up to 8 hours. Most of that time, the model is waiting for results from commands it is executing. Codex commonly sleeps to wait for results. Omnara implements process exit notifications and watch_process functionality, which automatically reinvokes the model after a process exits. This reduces idle token usage and the number of tool calls substantially.
2. Most exec output is deferred: Tool results carry at most 16KiB, which is about half of Claude Code. This helps limit input tokens for large commands. These outputs can always be searched through or fetched later, which is often faster than all of that extra context immediately.

As a result, Omnara produces more efficient tool calls than Codex and Claude. The agent no longer has to poll substantially for longer work, and more context is omitted by default. You can concretely see the payoff of this tool architecture in tasks both harnesses solve, such as in embedding-drift-monitor and fp8-rmsnorm-gemm. These benefits result in 11% cost savings versus Codex, around 5% versus Claude Code, and 12% faster task completion versus Claude Code.

What’s next? #

The labs build great models, and a high-performing harness paired with a reliable platform are force multipliers for those models. We at Omnara are always iterating on our tool architecture, and strive to be the best place for you to deploy your agents.

Every trace from this run is in our results viewer, and the harness code is on GitHub. If you want this efficiency for your own agents, with any model and on any machine, try Omnara today.

Footnotes #

1. We chose to use Modal for our Terminal-Bench evals, but Omnara supports 12 different sandbox providers and bring your own machine. ↩
2. For Claude, we excluded bio and security tasks which the model refused in most cases. For both models, we also excluded one task whose reference solution fails. For Opus, we omitted 12 tasks: 5 security tasks, 6 other refusals, and the broken-reference task. We also excluded one Opus attempt at rs-archive-clone that ended in a refusal. For Sol, we only omitted the broken reference task. ↩
3. MCPs or skills were attached to the agent config on demand for the tasks which required them ↩
4. We pulled the Claude Code benchmark from the TerminalBench leaderboard to compare against. One of our three Omnara Opus runs used an older version of Omnara.↩
5. We reran Terminal-Bench 4 with Codex 0.166.0, a more efficient version compared to the one benchmarked on the tbench.ai leaderboard (submitted September 29, 2026) ↩
6. Notably, we haven’t tested a code mode implementation in our harness yet. We found that our non-code mode implementation is already more efficient than Codex, which utilizes code mode natively. ↩