Everyone has an opinion about which AI coding model is best. Most of those opinions come from toy prompts or leaderboards. I wanted to see what happens on a real bug in a real open source repo, with tests the models cannot see.
So I gave the same bug to 9 free models at the same time and graded every result the same way.
The repo is pydynox, a Python DynamoDB library. Issue #560: the MemoryBackend test helper works as a decorator, but on an async function it exits its context before the coroutine runs. So the test runs against the real backend instead of the in-memory one. It is a small, realistic bug: easy to describe, easy to get subtly wrong.
opencode run, one model per run.
8 of 9 models produced a fix that passed the hidden tests. Big Pickle and Pulse Alpha both finished in 84 seconds. DeepSeek V4.1 Flash took 2 minutes 11 seconds. The rest took between 4 and 6.5 minutes. Total model cost: $0.
Speed was not the story, though. The failures were.
Crouton Alpha gave up after 10 seconds. No fix, no tests. That is the easy failure, because you notice it.
Nemotron 3.5 Lightning passed, but deleted an existing test. Its fix was correct and the hidden tests passed. It also removed a test that was already in the file. A green test run would never tell you that. Only reading the diff does.
That is the failure I worry about most with coding agents. Not "it did not work", but "it worked, and quietly changed something you did not ask for".
One bug is a small sample, so I ran a different kind of task. awslabs/stickler #517 asks for two classes to share their state handling through one private base class. The repo has 4,091 tests, so a refactor has nowhere to hide.
I gave it to 5 free models. All 5 kept the full suite green without editing a single test. Big Pickle was fastest again, at 4 minutes 34 seconds.
One model, Flint 40B, also ran ruff --fix across 8 Jupyter notebooks that had nothing to do with the task. Its diff touched 11 files. The others touched 3. Again: the tests passed, and the change was still not one you would merge.
For comparison I gave the first bug to three paid models: GLM 5.3 Flash, Kimi K3.1 Pro and Qwen 3.8 Max. All three fixed it, with tests that fail without the fix. They were not faster than the best free models.
There was one real quality difference. Only Qwen detected coroutine functions with inspect.iscoroutinefunction. The other two used asyncio.iscoroutinefunction, which is deprecated from Python 3.14. Both pass today's tests. Only one is still right next year. No test suite catches that. A reviewer who knows the language does.
One more lesson from the open source side: by the time I had a fix for one issue, a human contributor had claimed it 6 minutes earlier. Check the issue again right before you post.
I now use the same workflow for real contributions. Each pull request is reviewed and tested by me and labelled as AI-assisted, which is what the maintainers ask for.
Caveats: one run per model, on two tasks. Free models in this catalogue change often, so treat the names as a snapshot from November 2026, not a ranking.