Vibes Are Not a Benchmark: A 30-Minute Harness to Test Whether a Free Coding Model Can Touch Your Repo

Last month I wrote about the six questions I ask every new model. Those questions are good for a first impression, but they have a blind spot: they tell me how a model talks about code, not whether it can survive contact with my code. So I built a small, ugly, reproducible harness that runs any coding model against five tasks drawn from my own git history, scores the results with my real test suite, and writes the outcome to a CSV. This post is that harness, the reasoning behind it, and where free-tier tooling fits without falling apart.

Public benchmarks (HumanEval-style) have two problems for day-to-day tool selection. First, they're saturated and contaminated — models have seen them. Second, they don't look like your work. My work is: small refactors in a Python service, tests that fail for boring reasons, and occasional SQL migrations. So instead of asking "is this model smart," I ask "does this model reduce the time I spend on the five task shapes I actually do."

The trick that makes this cheap: your git history is a labeled dataset. Every commit that fixed a bug is a task (reproduce the fix) with a built-in grader (the tests that existed at the parent commit, plus the test added in the fix commit, if any).

Step 1: mine five candidate tasks from git.

for sha in $(git log --oneline -300 --grep='fix' -i --format='%h'); do
  if git show --name-only --format='' $sha | grep -q 'test'; then
    echo "$sha $(git log -1 --format='%s' $sha)"
  fi
done | head -20

I pick five where the diff is under 60 lines. Small diffs keep the test honest — you're measuring one decision, not endurance.

Step 2: for each chosen commit, build a task directory.

#!/usr/bin/env bash
set -euo pipefail
SHA=$1; NAME=$2
mkdir -p "tasks/$NAME"
git worktree add "tasks/$NAME/repo" "${SHA}^" 2>/dev/null
git show "$SHA" -- . ':!*test*' > "tasks/$NAME/gold.patch"
git show "$SHA" -- '*test*' > "tasks/$NAME/grading_test.patch"
git log -1 --format='%B' "$SHA" > "tasks/$NAME/prompt.txt"

Now each task has: a repo checked out before the fix, a problem statement (the original commit message), and a grader (the test that arrived with the fix). Nothing is hand-written.

Step 3: run the model and grade.

import subprocess, json, sys, pathlib

def run(cmd):
    return subprocess.run(cmd, shell=True, capture_output=True, text=True)

result = {"applies_clean": False, "tests_pass": False, "overreach": False}

patch = pathlib.Path("../model.patch")
if run(f"git apply --check {patch}").returncode == 0:
    result["applies_clean"] = True
    run(f"git apply {patch}")
    run("git apply ../grading_test.patch")  # the hidden test
    t = run("python -m pytest -x -q")
    result["tests_pass"] = (t.returncode == 0)
    gold_files = set(run("git apply --numstat ../gold.patch").stdout.split())
    changed = run("git diff --name-only").stdout.split()
    result["overreach"] = len(set(changed) - set(gold_files)) > 1

print(json.dumps(result))

Three scores per task, five tasks, fifteen data points. That's it. I deliberately score overreach

because in my experience the most expensive model failure isn't a wrong answer — it's a confident model quietly reformatting three unrelated files in the same patch.

I ran this against two setups: my usual paid tool, and a free model served through MonkeyCode, which currently offers free model access plus a free server option for exactly this kind of side experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I picked it for the comparison run because the experiment's entire premise is "is free good enough," and a zero-cost server meant I could rerun the harness as many times as I wanted without watching a meter. I'm not going to quote numbers as if they're universal — your repo is not my repo — but the shape of the result was consistent:

Two honest observations. First, the free model was fine on the unambiguous tasks — which, embarrassingly, are most of my real tickets. Second, both setups failed the ambiguous-prompt task, which confirmed the harness is measuring task clarity as much as model quality. A task neither setup can solve is a bad task, not a bad model; I rewrote that prompt and both passed.

Use this if you're evaluating coding assistants and you're tired of demo-driven decisions. Skip it if your work is greenfield-heavy — the whole method depends on having a fix-history to mine — or if you need statistically meaningful numbers for a procurement decision, in which case you need a bigger labeled set and more discipline than a bash script gives you.

What changed for me: the question stopped being "which model is best" and became "which tasks am I still doing by hand that a free model already handles." The harness took an afternoon; the answer paid for it in a week. If you want to replicate it, the three scripts above are the whole thing — point them at your own history, and if you need somewhere free to run the candidate model, that's a reasonable first use of MonkeyCode's free tier before you commit to anything.

I'd be curious what task shapes other people's git histories turn up — mine were depressingly dominated by off-by-one errors.