Vibes Are Not a Benchmark: A 30-Minute Harness to Test Whether a Free Coding Model Can Touch Your Repo

A developer built a 30-minute harness that tests coding models against five tasks mined from their own git history, scoring results with their real test suite. The harness, which runs any coding model and outputs a CSV, was used to compare a paid tool against a free model served through MonkeyCode, which sponsored the experiment. The developer found that public benchmarks are saturated and don't reflect real work, so they created a reproducible method using git commits as labeled datasets.

Last month I wrote about the six questions I ask every new model. Those questions are good for a first impression, but they have a blind spot: they tell me how a model talks about code, not whether it can survive contact with my code. So I built a small, ugly, reproducible harness that runs any coding model against five tasks drawn from my own git history, scores the results with my real test suite, and writes the outcome to a CSV. This post is that harness, the reasoning behind it, and where free-tier tooling fits without falling apart. Public benchmarks HumanEval-style have two problems for day-to-day tool selection. First, they're saturated and contaminated — models have seen them. Second, they don't look like your work. My work is: small refactors in a Python service, tests that fail for boring reasons, and occasional SQL migrations. So instead of asking "is this model smart," I ask "does this model reduce the time I spend on the five task shapes I actually do." The trick that makes this cheap: your git history is a labeled dataset. Every commit that fixed a bug is a task reproduce the fix with a built-in grader the tests that existed at the parent commit, plus the test added in the fix commit, if any . Step 1: mine five candidate tasks from git. Find bugfix-ish commits that also touched test files for sha in $ git log --oneline -300 --grep='fix' -i --format='%h' ; do if git show --name-only --format='' $sha | grep -q 'test'; then echo "$sha $ git log -1 --format='%s' $sha " fi done | head -20 I pick five where the diff is under 60 lines. Small diffs keep the test honest — you're measuring one decision, not endurance. Step 2: for each chosen commit, build a task directory. bash /usr/bin/env bash make task.sh <fix commit sha <task name set -euo pipefail SHA=$1; NAME=$2 mkdir -p "tasks/$NAME" git worktree add "tasks/$NAME/repo" "${SHA}^" 2 /dev/null The 'gold' patch and the grading test come from the fix commit itself git show "$SHA" -- . ': test ' "tasks/$NAME/gold.patch" git show "$SHA" -- ' test ' "tasks/$NAME/grading test.patch" Prompt = commit message of the fix, i.e. the problem statement git log -1 --format='%B' "$SHA" "tasks/$NAME/prompt.txt" Now each task has: a repo checked out before the fix, a problem statement the original commit message , and a grader the test that arrived with the fix . Nothing is hand-written. Step 3: run the model and grade. grade.py — run from tasks/<name /repo after applying the model's patch import subprocess, json, sys, pathlib def run cmd : return subprocess.run cmd, shell=True, capture output=True, text=True result = {"applies clean": False, "tests pass": False, "overreach": False} patch = pathlib.Path "../model.patch" if run f"git apply --check {patch}" .returncode == 0: result "applies clean" = True run f"git apply {patch}" run "git apply ../grading test.patch" the hidden test t = run "python -m pytest -x -q" result "tests pass" = t.returncode == 0 Overreach: did the model edit files outside the gold patch's scope? gold files = set run "git apply --numstat ../gold.patch" .stdout.split changed = run "git diff --name-only" .stdout.split result "overreach" = len set changed - set gold files 1 print json.dumps result Three scores per task, five tasks, fifteen data points. That's it. I deliberately score overreach because in my experience the most expensive model failure isn't a wrong answer — it's a confident model quietly reformatting three unrelated files in the same patch. I ran this against two setups: my usual paid tool, and a free model served through MonkeyCode, which currently offers free model access plus a free server option for exactly this kind of side experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I picked it for the comparison run because the experiment's entire premise is "is free good enough," and a zero-cost server meant I could rerun the harness as many times as I wanted without watching a meter. I'm not going to quote numbers as if they're universal — your repo is not my repo — but the shape of the result was consistent: | Task shape | Paid tool | Free model | |---|---|---| | One-line logic fix, existing failing test | pass | pass | | Off-by-one in pagination | pass | pass | | Refactor with hidden behavior requirement | pass | fail missed edge case | | SQL migration + backfill | pass | patch didn't apply cleanly | | Fix with ambiguous commit message | fail | fail | Two honest observations. First, the free model was fine on the unambiguous tasks — which, embarrassingly, are most of my real tickets. Second, both setups failed the ambiguous-prompt task, which confirmed the harness is measuring task clarity as much as model quality. A task neither setup can solve is a bad task, not a bad model; I rewrote that prompt and both passed. Use this if you're evaluating coding assistants and you're tired of demo-driven decisions. Skip it if your work is greenfield-heavy — the whole method depends on having a fix-history to mine — or if you need statistically meaningful numbers for a procurement decision, in which case you need a bigger labeled set and more discipline than a bash script gives you. What changed for me: the question stopped being "which model is best" and became "which tasks am I still doing by hand that a free model already handles." The harness took an afternoon; the answer paid for it in a week. If you want to replicate it, the three scripts above are the whole thing — point them at your own history, and if you need somewhere free to run the candidate model, that's a reasonable first use of MonkeyCode's free tier before you commit to anything. I'd be curious what task shapes other people's git histories turn up — mine were depressingly dominated by off-by-one errors.