Build a Personal Model Bake-Off: Testing Free AI Assistants on Your Real Bugs

A developer built a personal model bake-off to test free AI coding assistants on real bugs from a five-year-old codebase, rather than relying on generic leaderboards. The experiment used seven closed issues, blind grading, and a dependency-free Node scorer to evaluate models on compile success, test passing, and mergeability. The developer emphasized that the rubric must be defined before seeing results to avoid biased comparisons.

Last month I almost subscribed to an AI coding tool based on a comparison chart someone posted. Then I caught myself: that chart was built on competitive-programming puzzles and greenfield demos. My day job is gluing endpoints onto a five-year-old codebase with custom lint rules and a module nobody wants to touch. The chart couldn't tell me anything about that . So instead of picking a model from someone else's data, I ran a small bake-off on my own closed issues — using only free access, so the whole experiment cost nothing and committed me to nothing. This post is the recipe. The tooling is trivial; the discipline is the actual content. Public leaderboards answer a question I rarely have: "which model writes the best sorting algorithm from scratch?" My real questions are different: package.json ?None of those show up in generic evals, because they depend on context only I have. Which means the only benchmark that matters for the buying decision is one I run myself, on tasks where I already know what correct looks like. I pulled seven recently closed issues and turned each into a card: --- id: retry-regression-214 prompt: | The retry helper in src/net/retry.ts double-fires onAbort when the request times out. Fix it without changing the public API. context files: src/net/retry.ts, src/net/ tests /retry.test.ts known good: one-line guard on settled flag; all existing tests pass trap: models love rewriting the whole function and breaking test 4 --- Rules I set for the deck: The trap field is the most valuable part: I wrote down, in advance, how I expected models to fail. That turns vague impressions "model B felt sloppy" into checkable predictions. I ran each card against every model I could reach for free, saved each raw output to its own file, and — this part matters — renamed the files so I graded without knowing which model wrote what. Self-graded, unblinded comparisons are mostly astrology. Each graded output becomes one JSON record: { "task": "retry-regression-214", "output file": "out-07.patch", "compiles": true, "tests pass": false, "invented dependencies": , "review notes": "rewrote retry.ts wholesale; breaks test 4 as predicted", "would merge": false } Two of those fields must never be filled in by eye: compiles and tests pass come from actually executing the patch. I run that in a throwaway environment I wrote about a free sandbox harness in an earlier post so half-broken candidate code never touches my daily machine. Here's the whole scorer — dependency-free Node, deliberately boring: js // bakeoff.mjs — node bakeoff.mjs graded/ import { readFileSync, readdirSync } from 'node:fs'; import { join } from 'node:path'; const dir = process.argv 2 ?? 'graded'; // My priorities. Yours will differ — that is the feature. const score = r = r.compiles ? 1 : 0 + r.tests pass ? 3 : 0 + r.would merge ? 3 : 0 - 2 r.invented dependencies?.length ?? 0 ; const rows = readdirSync dir .filter f = f.endsWith '.json' .map f = JSON.parse readFileSync join dir, f , 'utf8' ; const perModel = new Map ; for const r of rows { const m = perModel.get r.output file.slice 0, 3 ?? ; m.push { task: r.task, score: score r } ; perModel.set r.output file.slice 0, 3 , m ; } for const model, entries of perModel { const avg = entries.reduce s, e = s + e.score, 0 / entries.length; const worst = entries.sort a, b = a.score - b.score 0 ; console.log ${model} avg=${avg.toFixed 2 } weakest=${worst.task} ${worst.score} ; } The weights are an argument with myself, written down: passing my tests and being mergeable count triple, a fabricated dependency is a hard penalty. Your weights should encode your pain — if you mostly want explanation quality, would merge shouldn't dominate. What matters is that the rubric exists before you look at results, not that it's perfect. The two things that usually make this annoying are needing several models to compare and needing a machine that isn't yours to run candidate code on. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode for both legs here: it currently offers free access to a selection of models and a free server option, which covered the full loop — same task deck pointed at multiple models, compile/test checks executed on their server, no credit card and nothing installed locally. The scorer above is indifferent to where outputs come from, though; pasting from any chat UI into the JSON files works identically. One deliberate omission: I recorded nothing about speed. Free-tier latency shifts with load and provider decisions, so a number I published would age badly — my rubric judges patch quality only. The headline wasn't a winner. It was three quieter findings: npm install . That check now lives permanently in my rubric; a public benchmark would never have surfaced it against tests pass said yes, would merge said no. Keep both columns; CI alone would have crowned that output.Three cards, two models, one afternoon, zero spend. If you want a no-cost place to run the loop, MonkeyCode's free model access plus free server covers it end to end — but the deck, the blinded grading, and the explicit weights are the real artifact, and they travel with you to any provider. Build the benchmark that knows your codebase, because nobody else's does.