# Vibes Are Not a Benchmark: A 30-Minute Harness to Test Whether a Free Coding Model Can Touch Your Repo > Source: > Published: 2026-08-12 23:09:08+00:00 Last month I wrote about the six questions I ask every new model. Those questions are good for a first impression, but they have a blind spot: they tell me how a model *talks* about code, not whether it can survive contact with *my* code. So I built a small, ugly, reproducible harness that runs any coding model against five tasks drawn from my own git history, scores the results with my real test suite, and writes the outcome to a CSV. This post is that harness, the reasoning behind it, and where free-tier tooling fits without falling apart. Public benchmarks (HumanEval-style) have two problems for day-to-day tool selection. First, they're saturated and contaminated — models have seen them. Second, they don't look like your work. My work is: small refactors in a Python service, tests that fail for boring reasons, and occasional SQL migrations. So instead of asking "is this model smart," I ask "does this model reduce the time I spend on the five task shapes I actually do." The trick that makes this cheap: your git history is a labeled dataset. Every commit that fixed a bug is a task (reproduce the fix) with a built-in grader (the tests that existed at the parent commit, plus the test added in the fix commit, if any). Step 1: mine five candidate tasks from git. ``` # Find bugfix-ish commits that also touched test files for sha in $(git log --oneline -300 --grep='fix' -i --format='%h'); do if git show --name-only --format='' $sha | grep -q 'test'; then echo "$sha $(git log -1 --format='%s' $sha)" fi done | head -20 ``` I pick five where the diff is under 60 lines. Small diffs keep the test honest — you're measuring one decision, not endurance. Step 2: for each chosen commit, build a task directory. ``` bash #!/usr/bin/env bash # make_task.sh set -euo pipefail SHA=$1; NAME=$2 mkdir -p "tasks/$NAME" git worktree add "tasks/$NAME/repo" "${SHA}^" 2>/dev/null # The 'gold' patch and the grading test come from the fix commit itself git show "$SHA" -- . ':!*test*' > "tasks/$NAME/gold.patch" git show "$SHA" -- '*test*' > "tasks/$NAME/grading_test.patch" # Prompt = commit message of the fix, i.e. the problem statement git log -1 --format='%B' "$SHA" > "tasks/$NAME/prompt.txt" ``` Now each task has: a repo checked out *before* the fix, a problem statement (the original commit message), and a grader (the test that arrived with the fix). Nothing is hand-written. Step 3: run the model and grade. ``` # grade.py — run from tasks//repo after applying the model's patch import subprocess, json, sys, pathlib def run(cmd): return subprocess.run(cmd, shell=True, capture_output=True, text=True) result = {"applies_clean": False, "tests_pass": False, "overreach": False} patch = pathlib.Path("../model.patch") if run(f"git apply --check {patch}").returncode == 0: result["applies_clean"] = True run(f"git apply {patch}") run("git apply ../grading_test.patch") # the hidden test t = run("python -m pytest -x -q") result["tests_pass"] = (t.returncode == 0) # Overreach: did the model edit files outside the gold patch's scope? gold_files = set(run("git apply --numstat ../gold.patch").stdout.split()) changed = run("git diff --name-only").stdout.split() result["overreach"] = len(set(changed) - set(gold_files)) > 1 print(json.dumps(result)) ``` Three scores per task, five tasks, fifteen data points. That's it. I deliberately score `overreach` because in my experience the most expensive model failure isn't a wrong answer — it's a confident model quietly reformatting three unrelated files in the same patch. I ran this against two setups: my usual paid tool, and a free model served through MonkeyCode, which currently offers free model access plus a free server option for exactly this kind of side experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I picked it for the comparison run because the experiment's entire premise is "is free good enough," and a zero-cost server meant I could rerun the harness as many times as I wanted without watching a meter. I'm not going to quote numbers as if they're universal — your repo is not my repo — but the *shape* of the result was consistent: | Task shape | Paid tool | Free model | |---|---|---| | One-line logic fix, existing failing test | pass | pass | | Off-by-one in pagination | pass | pass | | Refactor with hidden behavior requirement | pass | fail (missed edge case) | | SQL migration + backfill | pass | patch didn't apply cleanly | | Fix with ambiguous commit message | fail | fail | Two honest observations. First, the free model was fine on the unambiguous tasks — which, embarrassingly, are most of my real tickets. Second, both setups failed the ambiguous-prompt task, which confirmed the harness is measuring task clarity as much as model quality. A task neither setup can solve is a bad task, not a bad model; I rewrote that prompt and both passed. Use this if you're evaluating coding assistants and you're tired of demo-driven decisions. Skip it if your work is greenfield-heavy — the whole method depends on having a fix-history to mine — or if you need statistically meaningful numbers for a procurement decision, in which case you need a bigger labeled set and more discipline than a bash script gives you. What changed for me: the question stopped being "which model is best" and became "which tasks am I still doing by hand that a free model already handles." The harness took an afternoon; the answer paid for it in a week. If you want to replicate it, the three scripts above are the whole thing — point them at your own history, and if you need somewhere free to run the candidate model, that's a reasonable first use of MonkeyCode's free tier before you commit to anything. I'd be curious what task shapes other people's git histories turn up — mine were depressingly dominated by off-by-one errors.