The benchmark win that doesn’t survive contact with code #
Open models, including Kimi, GLM, DeepSeek, and Qwen, frequently post impressive benchmark scores, sometimes even matching or surpassing frontier models like Anthropic’s Claude Opus and OpenAI’s Project Project Astra. These wins, however, are often context-dependent, reflecting performance on specific, self-contained coding challenges rather than the full spectrum of software development.
A crucial distinction lies in task complexity and duration. Open models excel at short, discrete tasks where strong reasoning can be applied to isolated problems. This makes them highly effective for many business automations and individual workflow steps.
Real-world software engineering, conversely, demands an agent’s ability to maintain context, adhere to evolving instructions, and preserve architectural decisions across numerous tool calls and iterative refinements. This long-horizon consistency is where open models currently fall short.
While benchmarks offer valuable signals, they don't fully capture end-to-end coding proficiency. Resources like SWE-bench provide a more comprehensive evaluation of a model's software engineering capabilities, highlighting the gap between isolated task performance and sustained project work.
Where open models actually earn their keep #
Open models do, however, earn their keep in more bounded contexts. For specific, contained tasks—drafting a function, transforming a schema, or generating unit tests—these models can offer quality comparable to frontier models, according to Medin’s practical experience. This parity extends to many routine steps within business automations.
Businesses frequently leverage open models to control spend and maintain data privacy by running them locally or through cost-effective APIs. A top-tier model like Project Project Astra or Claude Opus 5.5 isn't always necessary for tasks such as:
- Data validation
- Simple content generation
- Internal API routing logic
Critically, cost isn't a simple per-token comparison. The true price of a successful outcome includes reliability, the frequency of retries, the human effort required for review, and the potential cost of a failed change. While open models might seem cheaper by raw token count, their contextual limitations in complex, long-horizon tasks can lead to higher overall expenses due to increased iteration and correction. This nuance is crucial for effective resource allocation.
The long-run failure: drift, retries, and lost rules #
Open models falter dramatically when coding workflows expand in scope. Medin observes that they frequently lose track of repository conventions, disregard established constraints, and become entangled in costly review-and-fix cycles. This "long-run failure" stems from an inability to maintain attention to a growing body of rules and context.
To quantify this, Medin conducted a controlled experiment building a Northstar game. He assigned 12 feature tickets to three models: Claude, OpenAI, and Kimi. Claude completed all 12 tickets in approximately 14 hours. OpenAI finished 11 tickets, but required about 28 hours. Kimi, by contrast, managed only 5 tickets over 44 hours.
This comparison, while a single creator’s test and not a universal leaderboard, powerfully illustrates how instruction adherence and iteration overhead can shape project outcomes. Open models often incur higher overall costs due to repeated token expenditure on corrections and retries, despite lower per-token pricing. For further exploration of benchmark methodologies, consider the SWE-bench Official Benchmark and Leaderboard.
This performance gap suggests a fundamental difference in how models handle long-horizon tasks. While open models can excel at isolated code generation, they struggle with the sustained contextual awareness necessary for complex software development, where a model must integrate changes across a large codebase while adhering to evolving constraints and design principles.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
A hybrid workflow beats an all-or-nothing bet #
Rather than an all-or-nothing bet, a hybrid workflow offers immediate utility. Route discrete, well-scoped tasks—like drafting a function or transforming a schema—to open models, leveraging their cost efficiency and competitive performance on bounded problems. Reserve frontier systems such as Claude Opus for complex architectural planning, high-stakes review, and changes that span a large codebase.
Keep the harness explicit and robust. Break work into smaller tickets, enforce coding conventions with automated tests and linting, and continuously measure key metrics. Track completion rates, retry counts, review failures, and the cost per merged change to objectively assess model performance and efficiency.
Open models are already practical, but benchmark parity does not guarantee they can run an entire software factory. Teams should diligently test models against their own workflows, shifting responsibility as reliability demonstrably improves. This pragmatic, data-driven approach allows for strategic integration of open-source innovation while mitigating the risks of long-horizon drift and inconsistent execution.
Frequently Asked Questions #
Why can open models score well on coding benchmarks but struggle in real projects?
Benchmarks often test bounded tasks, while real projects require agents to follow rules, retain context, and make consistent decisions across many steps.
Are open-weight models useful for AI coding?
Yes. They can perform well on discrete, short tasks and routine workflow steps, especially when the task is clearly scoped.
What is a long-horizon coding task?
It is work that spans many steps, such as planning, editing files, running tests, responding to reviews, and maintaining consistency across a codebase.
How can teams use open models without relying on them for everything?
Use open models for well-defined, lower-risk tasks, and reserve frontier models for complex planning, review, and work that demands sustained context.