Zach Lloyd is turning past agent runs into model evaluations and routing rules, with an internal cost claim that still needs outside proof.
By RuntimeWire Staff · Published
Primary source: Warp on X
Why it matters #
Coding-agent costs depend on a team's code, tools and task mix. Warp is selling the measurement and routing layer that can turn rapid model churn into an operating advantage.
Zach Lloyd (@zachlloydtweets) launched Factory Benchmarks inside Warp's agent infrastructure on September 3rd, giving engineering teams a way to compare coding models against their own historical tasks and route future work based on the results.
Lloyd started Warp in 2020 to rebuild a terminal interface that he believed had barely changed in four decades. According to Warp's team profile, the former Google principal engineer ran engineering for Google Sheets and the broader Google Docs suite before becoming CTO of TIME and co-founding SelfMade. He has since expanded Warp's original thesis considerably. Warp now wants to provide the control plane for fleets of coding agents, including the system that decides which model should touch each ticket.
Factory Benchmarks supplies the measurement layer for that bet. Warp detailed the launch in a September 3rd product post, one day after Lloyd published a detailed internal case study called WarpBench.
The timing follows Warp's August 18th introduction of Warp Factories, an early-access product for running coding agents across steps such as triage, implementation and review. RuntimeWire reported in August that Warp had also separated its coding agent from Warp Terminal, making the agent available from other command-line environments. Factory Benchmarks pushes Warp another step away from its terminal origins and toward infrastructure that sits above individual agents and models.
A private benchmark built from production work
Teams can create a benchmark by selecting previous agent runs stored in Warp Factories or asking an agent, through the Warp Factories MCP, to assemble a representative task set. Warp replays each task from its original code and Git state while holding the relevant factory configuration constant.
Users then choose the models, harnesses and configurations they want to compare. Built-in scorers cover correctness, code quality, efficiency, verbosity and cost. Teams can write rubrics for requirements that public coding benchmarks rarely capture, including adherence to a Figma mockup or the quality of end-to-end tests.
Each benchmark executes a matrix of tasks and configurations. Warp's report recommends an overall configuration, breaks down performance task by task and plots tradeoffs such as cost against correctness. The results can feed into model-routing rules, allowing a factory to send one category of work to a cheaper model and reserve a more expensive model for tasks where it earns the difference.
That last step is the commercial center of the launch. Model providers publish benchmark results that help sell their newest releases, while coding-agent vendors promote results produced with their own harnesses. Warp is betting that engineering leaders will pay for an evaluation tied to their repositories, prompts, tools and acceptance criteria.
Lloyd argues in the WarpBench case study that public task sets can be poor proxies for production work and may be contaminated by inclusion in model training data. The first point follows directly from the variation among private codebases. The contamination claim is Lloyd's assessment, and Warp has not published an independent analysis covering the public benchmarks it references.
Warp's $2,130 model test
Warp built its internal benchmark from 30 historical tasks across Go, React and Rust repositories. The sample included interface changes, database work, backend logic and systems programming, with tasks divided into approximate S, M, L and XL scopes.
Warp tested Opus 5, GPT-5 Sol, Gemini 3.7, Grok 4.6 and GLM 5.3 Flash using Warp Agent as the harness. The run took 3 hours and 46 minutes and cost $2,130.57, according to Lloyd. Setup took less than 30 minutes, Warp says.
The expense helps define the product's likely buyer. This is an evaluation system for teams already spending enough on coding agents that a four-figure experiment can produce a measurable return. Warp itself recommends rerunning benchmarks when models or important factory inputs change, rather than running them continuously.
The WarpBench case study says Warp replaced its "auto (genius)" routed configuration, which sent complex tasks most frequently to Opus 5, with Grok 4.6 High as the primary implementation agent after the benchmark found comparable quality at lower cost. Warp says an earlier internal test cut median cost per completed pull request from about $80 to $30 while keeping merge rate constant; the company has not independently validated that result.
Warp's case study says the benchmark produced a 63% cost reduction, but the result remains an internal company report rather than an audited comparison. The arithmetic behind that claim is a 62.5% reduction after rounding the two reported figures.
At a sustained saving of $50 per completed pull request, the $2,130.57 benchmark would recover its direct run cost after about 43 completed pull requests. That calculation excludes the cost of setup, reviewing results, future benchmark runs and any differences between the sampled tasks and later production work.
Warp also reported that its internal task-compliance score rose from 69% to 87%. Those scores were generated by Warp's own LLM-as-a-judge system rather than an independent evaluator. Warp's underlying chart showed median cost falling from $81.13 on August 26th to $19.06 on August 30th, a sharper decline than the rounded $80-to-$30 headline. The chart covers specific dates, while the broader claim describes completed pull requests over a less precise period, so the figures should not be treated as one controlled measurement. Warp later selected GPT-5.6 Sol as its default implementation model after expanding the benchmark. Lloyd estimated another 25% improvement while explicitly noting variance. That projected gain had not yet been established over a longer production period when he published the case study.
The harness comparison is still coming
Factory Benchmarks can vary models today, but Warp says direct cross-harness comparisons involving Warp Agent, Claude Code and Codex are not yet fully supported. That matters because model results can change with the surrounding agent: its tools, prompts, context management and implementation loop all affect performance.
Until those comparisons arrive, WarpBench mainly demonstrates model selection inside Warp's own harness. The feature remains useful for teams already committed to Warp Factories, but it does not yet provide a neutral bake-off among the coding-agent products engineering leaders are choosing between.
Warp is offering qualified early-access customers up to $10,000 in Factory usage through its access program. That credit also gives Warp a practical way to collect more benchmark configurations and refine a product whose strongest evidence currently comes from Warp's own repositories.
Lloyd's broader strategy follows a June 18th memo to Warp employees, in which he described engineers as "factory engineers" whose job is to build the system that builds the product. Factory Benchmarks turns that operating doctrine into a product: preserve the runs, replay the work, measure the result and encode the winner into routing rules.
The durable part of Warp's pitch is model turnover itself. A team that standardizes on one coding model will have to revisit that decision each time a lab ships a cheaper or more capable release. Lloyd is building Warp around that recurring evaluation cycle, positioning the platform to benefit regardless of which model wins the next test.