Evaluating Open Weight Models for Software Maintenance

Bosun's evaluation of open-weight models for software maintenance found that GLM 5.2 produced a high-quality pull request at the lowest cost of $4.62, outperforming OpenAI's GPT-5.6 Terra baseline at $5.89. Kimi K3 was cheapest at $4.43 but cheated the workflow, while Qwen 3.7 Max's PR was not mergeable, and Nemotron 3 Ultra was most expensive at $8.51.

All posts / Evaluating Open Weight Models for Software Maintenance Can open weight models automate software maintenance effectively and at lower cost? In this article we’ll have some fun and evaluate open-weight models by utilizing them in a software maintenance workflow run. We’ll use GPT-5.6 as a baseline and then compare it against GLM 5.2, Kimi K3, Qwen 3.8 Max and Nemotron 3 Ultra. Which model do you think is going to be the cheapest? Which model is going to have the most difficulty coming up with good pull requests? Think of the answer and then scroll to the bottom to check your preconceived notions. If you’re interested in how these numbers were established, continue reading Two major events shook the agentic coding industry last month. The first was the US government blocking the access to state of the art models, suggesting not just that capabilities of models are starting to have impact on existing IT infrastructure but also that the US is government is not afraid to take away access to models when they see fit. And even more dangerously, they suggested to take away access from entities outside the US specifically. The other event was the release of Chinese built open weight models Kimi K3 and GLM 5.2 which very openly challenge the status quo of Anthropic and OpenAI’s state of the art hegemony. If for no other reason than to be prudent, we should be evaluating these new models for use in coding workflows. At Bosun we strive to automate software maintenance. We believe that we can help companies stay agile and focused by taking away distractions that emerge from codebases as they age. These events initially sparked two questions and then some further interesting follow up questions that we’ll dive into in this series of articles. The initial questions are: - Can we effectively automate software maintenance with sovereign models? - Can we save on spending by using open weight models? To answer these questions we picked one of our workflows and did a series of experiments. To establish a starting point we did no preparation at all and simply ran the workflow with OpenAI’s GPT-5.6 as a baseline, and then a set of the state of the art open weight models to compare performance. The workflow we’ll be running is a simple but thorough workflow that helps companies upgrade their legacy as in PHP7 era PHP codebase, specifically to ensure they can continue to address security concerns. In a later article we’ll dive a bit deeper into how the workflow works, for now it’s enough to know that it consists of a sequence of simple steps that ensure a pull request is generated that bumps the dependencies of a PHP project to the next major version that has no outstanding security advisories while ensuring behavioral tests keep passing. | Model | Input | Output | Est. cost | Churn | |---|---|---|---|---| | GPT-5.6 Terra | 10,524,721 | 134,783 | $5.89 | 3,572 | | GLM 5.2 | 19,333,086 | 276,136 | $4.62 | 5,102 | | Kimi K3 | 5,390,750 | 92,148 | $4.43 | 5,029 | | Qwen 3.7 Max | 13,179,162 | 424,604 | $7.71 | 5,079 | | Nemotron 3 Ultra | 24,499,222 | 182,583 | $8.51 | 4,633 | The results were surprising to me. The nature of software maintenance is that it requires less creativity than the development of new features does, so I expected the cheaper and probably less capable models to outperform OpenAI if by nothing else on cost, however the savings were a lot smaller than expected which was mostly due to the open weight models using a lot more tokens. Kimi K3 discovered a cheat its way through the workflow, reducing the complexity of the work it had to perform. Similarly, Qwen produced a PR that passed through the workflow gates but upon careful review was not of mergeable quality. GLM is the standout success, its PR was of high quality and despite using a lot of tokens it eked out a definite win on cost. Judging these pull requests on quality is a bit tricky, we’ll treat that aspect in a future post, but for now it’s important to note that GPT-5.6 terra produced the highest quality PR with the lowest amount of code churn and with great token efficiency. This baseline established that interesting things are definitely going on, but we can’t quite call this a definite result since the workflow was created by iterating using OpenAI models. To reduce the bias of the workflow towards OpenAI, I had an agent list what parts of the workflow each model was struggling with. This was a very useful exercise as it not only resulted in a list of improvements to the workflow, it also laid bare some bugs that were having an outsized impact on the open weight models. Note that I could have opted to not show the previous run because it included bugs, but I think it revealed something important when considering making your system work with less capable models. The models actions are a lot more noisy, and the noise they create can really shake out any bugs you have in your system. Just running the workflow with less capable models and then fixing all the reliability and consistency problems that arose from that made our platform significantly more robust. | Model | Input | Output | Est. cost | Churn | |---|---|---|---|---| | GPT-5.6 Terra | 6,887,905 | 196,879 | $4.35 | 3,722 | | GLM-5.2 | 24,030,876 | 396,610 | $2.78 | 4,834 | | Kimi-K3 | 5,345,768 | 175,929 | $5.62 | 5,022 | | Qwen 3.7 Max | 10,871,657 | 341,078 | $6.51 | 4,691 | | Nemotron 3 Ultra | 16,070,699 | 112,831 | $4.23 | 2,187 | Fixing the outright bugs reduced the costs and improved the results for all models. Especially GLM now pulls ahead of the pack, still doing significantly more reads and writes, but due to its cost per token being so much lower it was over 30% cheaper than before. It turned out Kimi had cheated in its prior test, and it didn’t get away with it this run so even though it’s still performing well it’s not as cheap as it was in the first run. Now that the equipment was made sturdy, the next step was evening the playing field. I went through each of the workflow suggestions that arose from the history analysis and implemented the improvements. | Model | Input | Output | Est. cost | Churn | |---|---|---|---|---| | GPT‑5.6 Terra | 5,675,859 | 149,670 | €4.27 | 4,519 | | GLM 5.2 | 11,535,925 | 225,123 | €2.31 | 3,912 | | Kimi K3 | 3,585,049 | 135,484 | €5.37 | 3,941 | | Qwen 3.8 Max | 9,423,716 | 617,530 | €12.44 | 3,998 | | Nemotron 3 Ultra 550B | 14,234,337 | 200,067 | €5.82 | 4,989 | Quite surprisingly although tightening up the workflow improved token efficiency across the board, it did not have a big impact on overall cost. GLM 5.2 takes the cost lead even further. These are some key takeaways I got from this last run: - GPT-5.6 is really aggressively priced for how capable the model is - GLM 5.2 is the most economical choice by far, despite being token inefficient by the shear cheapness of its tokens - Kimi K3 really does seem close to being on par with GPT-5.6 in capability per token, but at least on open router is more expensive to run - Nemotron Ultra is not super far behind, but really needed the extra instructions just to be able to complete the workflow successfully - Qwen 3.8 is middle of the pack, but its chattiness and the increased token cost over Qwen 3.7 made it the least economical choice in the final benchmark There’s loads more to learn from playing around with software maintenance workflows, open weight models and tweaking the parameters so I’ll turn this blog post in a mini-series, stay tuned if you want to learn more. If you’re interested in automating software maintenance please check out bosun.ai https://bosun.ai/ and sign up for a free beta account