{"slug": "build-a-personal-model-bake-off-testing-free-ai-assistants-on-your-real-bugs", "title": "Build a Personal Model Bake-Off: Testing Free AI Assistants on Your Real Bugs", "summary": "A developer built a personal model bake-off to test free AI coding assistants on real bugs from a five-year-old codebase, rather than relying on generic leaderboards. The experiment used seven closed issues, blind grading, and a dependency-free Node scorer to evaluate models on compile success, test passing, and mergeability. The developer emphasized that the rubric must be defined before seeing results to avoid biased comparisons.", "body_md": "Last month I almost subscribed to an AI coding tool based on a comparison chart someone posted. Then I caught myself: that chart was built on competitive-programming puzzles and greenfield demos. My day job is gluing endpoints onto a five-year-old codebase with custom lint rules and a module nobody wants to touch. The chart couldn't tell me anything about *that*.\n\nSo instead of picking a model from someone else's data, I ran a small bake-off on my own closed issues — using only free access, so the whole experiment cost nothing and committed me to nothing. This post is the recipe. The tooling is trivial; the discipline is the actual content.\n\nPublic leaderboards answer a question I rarely have: \"which model writes the best sorting algorithm from scratch?\" My real questions are different:\n\n`package.json`\n\n?None of those show up in generic evals, because they depend on context only I have. Which means the only benchmark that matters for the buying decision is one I run myself, on tasks where I already know what correct looks like.\n\nI pulled seven recently closed issues and turned each into a card:\n\n```\n---\nid: retry-regression-214\nprompt: |\n  The retry helper in src/net/retry.ts double-fires onAbort when\n  the request times out. Fix it without changing the public API.\ncontext_files: [src/net/retry.ts, src/net/__tests__/retry.test.ts]\nknown_good: one-line guard on settled flag; all existing tests pass\ntrap: models love rewriting the whole function and breaking test #4\n---\n```\n\nRules I set for the deck:\n\nThe `trap`\n\nfield is the most valuable part: I wrote down, in advance, how I expected models to fail. That turns vague impressions (\"model B felt sloppy\") into checkable predictions.\n\nI ran each card against every model I could reach for free, saved each raw output to its own file, and — this part matters — renamed the files so I graded without knowing which model wrote what. Self-graded, unblinded comparisons are mostly astrology.\n\nEach graded output becomes one JSON record:\n\n```\n{\n  \"task\": \"retry-regression-214\",\n  \"output_file\": \"out-07.patch\",\n  \"compiles\": true,\n  \"tests_pass\": false,\n  \"invented_dependencies\": [],\n  \"review_notes\": \"rewrote retry.ts wholesale; breaks test #4 as predicted\",\n  \"would_merge\": false\n}\n```\n\nTwo of those fields must never be filled in by eye: `compiles`\n\nand `tests_pass`\n\ncome from actually executing the patch. I run that in a throwaway environment (I wrote about a free sandbox harness in an earlier post) so half-broken candidate code never touches my daily machine.\n\nHere's the whole scorer — dependency-free Node, deliberately boring:\n\n``` js\n// bakeoff.mjs — node bakeoff.mjs graded/\nimport { readFileSync, readdirSync } from 'node:fs';\nimport { join } from 'node:path';\n\nconst dir = process.argv[2] ?? 'graded';\n\n// My priorities. Yours will differ — that is the feature.\nconst score = (r) =>\n  (r.compiles ? 1 : 0) +\n  (r.tests_pass ? 3 : 0) +\n  (r.would_merge ? 3 : 0) -\n  2 * (r.invented_dependencies?.length ?? 0);\n\nconst rows = readdirSync(dir)\n  .filter((f) => f.endsWith('.json'))\n  .map((f) => JSON.parse(readFileSync(join(dir, f), 'utf8')));\n\nconst perModel = new Map();\nfor (const r of rows) {\n  const m = perModel.get(r.output_file.slice(0, 3)) ?? [];\n  m.push({ task: r.task, score: score(r) });\n  perModel.set(r.output_file.slice(0, 3), m);\n}\n\nfor (const [model, entries] of perModel) {\n  const avg = entries.reduce((s, e) => s + e.score, 0) / entries.length;\n  const worst = entries.sort((a, b) => a.score - b.score)[0];\n  console.log(`${model}  avg=${avg.toFixed(2)}  weakest=${worst.task} (${worst.score})`);\n}\n```\n\nThe weights are an argument with myself, written down: passing my tests and being mergeable count triple, a fabricated dependency is a hard penalty. Your weights should encode *your* pain — if you mostly want explanation quality, `would_merge`\n\nshouldn't dominate. What matters is that the rubric exists before you look at results, not that it's perfect.\n\nThe two things that usually make this annoying are needing *several* models to compare and needing a machine that isn't yours to run candidate code on.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nI used MonkeyCode for both legs here: it currently offers free access to a selection of models and a free server option, which covered the full loop — same task deck pointed at multiple models, compile/test checks executed on their server, no credit card and nothing installed locally. The scorer above is indifferent to where outputs come from, though; pasting from any chat UI into the JSON files works identically. One deliberate omission: I recorded nothing about speed. Free-tier latency shifts with load and provider decisions, so a number I published would age badly — my rubric judges patch quality only.\n\nThe headline wasn't a winner. It was three quieter findings:\n\n`npm install`\n\n. That check now lives permanently in my rubric; a public benchmark would never have surfaced it against `tests_pass`\n\nsaid yes, `would_merge`\n\nsaid no. Keep both columns; CI alone would have crowned that output.Three cards, two models, one afternoon, zero spend. If you want a no-cost place to run the loop, MonkeyCode's free model access plus free server covers it end to end — but the deck, the blinded grading, and the explicit weights are the real artifact, and they travel with you to any provider. Build the benchmark that knows your codebase, because nobody else's does.", "url": "https://wpnews.pro/news/build-a-personal-model-bake-off-testing-free-ai-assistants-on-your-real-bugs", "canonical_source": "https://dev.to/codejs_8314/build-a-personal-model-bake-off-testing-free-ai-assistants-on-your-real-bugs-4abg", "published_at": "2026-08-10 08:12:10+00:00", "updated_at": "2026-08-10 08:47:16.915725+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models"], "entities": ["Node"], "alternates": {"html": "https://wpnews.pro/news/build-a-personal-model-bake-off-testing-free-ai-assistants-on-your-real-bugs", "markdown": "https://wpnews.pro/news/build-a-personal-model-bake-off-testing-free-ai-assistants-on-your-real-bugs.md", "text": "https://wpnews.pro/news/build-a-personal-model-bake-off-testing-free-ai-assistants-on-your-real-bugs.txt", "jsonld": "https://wpnews.pro/news/build-a-personal-model-bake-off-testing-free-ai-assistants-on-your-real-bugs.jsonld"}}