{"slug": "open-ai-models-hit-a-brutal-coding-wall", "title": "Open AI Models Hit a Brutal Coding Wall", "summary": "In a controlled test by creator Medin, Anthropic's Claude completed all 12 feature tickets for a Northstar game in about 14 hours, while OpenAI's model finished 11 tickets in roughly 28 hours and Kimi completed only 5 tickets over 44 hours. The comparison illustrates that open models such as Kimi, GLM, DeepSeek and Qwen can match frontier models on short, self-contained coding tasks but lose track of repository conventions and constraints on long-horizon work, driving up retries and overall cost despite lower per-token pricing. Medin notes open models still offer comparable quality for bounded tasks like drafting a function, transforming a schema or generating unit tests.", "body_md": "## The benchmark win that doesn’t survive contact with code\n\nOpen models, including [Kimi](https://www.stork.ai/en/kimik3), GLM, [DeepSeek](https://www.stork.ai/en/deepseek), and [Qwen](https://www.stork.ai/en/qwen), frequently post impressive benchmark scores, sometimes even matching or surpassing frontier models like Anthropic’s [Claude](https://www.stork.ai/en/poe-claude) Opus and OpenAI’s Project Project Astra. These wins, however, are often context-dependent, reflecting performance on specific, self-contained coding challenges rather than the full spectrum of software development.\n\nA crucial distinction lies in task complexity and duration. Open models excel at short, discrete tasks where strong reasoning can be applied to isolated problems. This makes them highly effective for many business automations and individual workflow steps.\n\nReal-world software engineering, conversely, demands an agent’s ability to maintain context, adhere to evolving instructions, and preserve architectural decisions across numerous tool calls and iterative refinements. This long-horizon consistency is where open models currently fall short.\n\nWhile benchmarks offer valuable signals, they don't fully capture end-to-end coding proficiency. Resources like **SWE-bench** provide a more comprehensive evaluation of a model's software engineering capabilities, highlighting the gap between isolated task performance and sustained project work.\n\n## Where open models actually earn their keep\n\nOpen models do, however, earn their keep in more bounded contexts. For specific, contained tasks—drafting a function, transforming a schema, or generating unit tests—these models can offer quality comparable to frontier models, according to Medin’s practical experience. This parity extends to many routine steps within business automations.\n\nBusinesses frequently leverage open models to control spend and maintain data privacy by running them locally or through cost-effective APIs. A top-tier model like Project Project Astra or Claude Opus 5.5 isn't always necessary for tasks such as:\n\n- Data validation\n- Simple content generation\n- Internal API routing logic\n\nCritically, cost isn't a simple per-token comparison. The true price of a successful outcome includes reliability, the frequency of retries, the human effort required for review, and the potential cost of a failed change. While open models might seem cheaper by raw token count, their contextual limitations in complex, long-horizon tasks can lead to higher overall expenses due to increased iteration and correction. This nuance is crucial for effective resource allocation.\n\n## The long-run failure: drift, retries, and lost rules\n\nOpen models falter dramatically when coding workflows expand in scope. Medin observes that they frequently lose track of repository conventions, disregard established constraints, and become entangled in costly review-and-fix cycles. This \"long-run failure\" stems from an inability to maintain **attention** to a growing body of rules and context.\n\nTo quantify this, Medin conducted a controlled experiment building a Northstar game. He assigned 12 feature tickets to three models: Claude, OpenAI, and Kimi. Claude completed all 12 tickets in approximately 14 hours. OpenAI finished 11 tickets, but required about 28 hours. Kimi, by contrast, managed only 5 tickets over 44 hours.\n\nThis comparison, while a single creator’s test and not a universal leaderboard, powerfully illustrates how instruction adherence and iteration overhead can shape project outcomes. Open models often incur higher overall costs due to repeated token expenditure on corrections and retries, despite lower per-token pricing. For further exploration of benchmark methodologies, consider the [SWE-bench Official Benchmark and Leaderboard](https://www.swebench.com).\n\nThis performance gap suggests a fundamental difference in how models handle **long-horizon tasks**. While open models can excel at isolated code generation, they struggle with the sustained contextual awareness necessary for complex software development, where a model must integrate changes across a large codebase while adhering to evolving constraints and design principles.\n\nEnjoying this? Get one like it in your inbox each morning.\n\none email a day · unsubscribe in two clicks · no third-party tracking\n\n## A hybrid workflow beats an all-or-nothing bet\n\nRather than an all-or-nothing bet, a **hybrid workflow** offers immediate utility. Route discrete, well-scoped tasks—like drafting a function or transforming a schema—to open models, leveraging their cost efficiency and competitive performance on bounded problems. Reserve frontier systems such as Claude Opus for complex architectural planning, high-stakes review, and changes that span a large codebase.\n\nKeep the harness explicit and robust. Break work into smaller tickets, enforce coding conventions with automated tests and linting, and continuously measure key metrics. Track completion rates, retry counts, review failures, and the **cost per merged change** to objectively assess model performance and efficiency.\n\nOpen models are already practical, but benchmark parity does not guarantee they can run an entire software factory. Teams should diligently test models against their own workflows, shifting responsibility as reliability demonstrably improves. This pragmatic, data-driven approach allows for strategic integration of **open-source innovation** while mitigating the risks of long-horizon drift and inconsistent execution.\n\n## Frequently Asked Questions\n\n### Why can open models score well on coding benchmarks but struggle in real projects?\n\nBenchmarks often test bounded tasks, while real projects require agents to follow rules, retain context, and make consistent decisions across many steps.\n\n### Are open-weight models useful for AI coding?\n\nYes. They can perform well on discrete, short tasks and routine workflow steps, especially when the task is clearly scoped.\n\n### What is a long-horizon coding task?\n\nIt is work that spans many steps, such as planning, editing files, running tests, responding to reviews, and maintaining consistency across a codebase.\n\n### How can teams use open models without relying on them for everything?\n\nUse open models for well-defined, lower-risk tasks, and reserve frontier models for complex planning, review, and work that demands sustained context.", "url": "https://wpnews.pro/news/open-ai-models-hit-a-brutal-coding-wall", "canonical_source": "https://www.stork.ai/blog/open-ai-models-hit-a-brutal-coding-wall", "published_at": "2026-11-05 04:02:35+00:00", "updated_at": "2026-11-05 04:49:29.814439+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-research", "ai-tools"], "entities": ["Kimi", "GLM", "DeepSeek", "Qwen", "Anthropic", "Claude", "OpenAI", "SWE-bench"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/open-ai-models-hit-a-brutal-coding-wall", "markdown": "https://wpnews.pro/news/open-ai-models-hit-a-brutal-coding-wall.md", "text": "https://wpnews.pro/news/open-ai-models-hit-a-brutal-coding-wall.txt", "jsonld": "https://wpnews.pro/news/open-ai-models-hit-a-brutal-coding-wall.jsonld"}}