{"slug": "vibes-are-not-a-benchmark-a-30-minute-harness-to-test-whether-a-free-coding-can", "title": "Vibes Are Not a Benchmark: A 30-Minute Harness to Test Whether a Free Coding Model Can Touch Your Repo", "summary": "A developer built a 30-minute harness that tests coding models against five tasks mined from their own git history, scoring results with their real test suite. The harness, which runs any coding model and outputs a CSV, was used to compare a paid tool against a free model served through MonkeyCode, which sponsored the experiment. The developer found that public benchmarks are saturated and don't reflect real work, so they created a reproducible method using git commits as labeled datasets.", "body_md": "Last month I wrote about the six questions I ask every new model. Those questions are good for a first impression, but they have a blind spot: they tell me how a model *talks* about code, not whether it can survive contact with *my* code. So I built a small, ugly, reproducible harness that runs any coding model against five tasks drawn from my own git history, scores the results with my real test suite, and writes the outcome to a CSV. This post is that harness, the reasoning behind it, and where free-tier tooling fits without falling apart.\n\nPublic benchmarks (HumanEval-style) have two problems for day-to-day tool selection. First, they're saturated and contaminated — models have seen them. Second, they don't look like your work. My work is: small refactors in a Python service, tests that fail for boring reasons, and occasional SQL migrations. So instead of asking \"is this model smart,\" I ask \"does this model reduce the time I spend on the five task shapes I actually do.\"\n\nThe trick that makes this cheap: your git history is a labeled dataset. Every commit that fixed a bug is a task (reproduce the fix) with a built-in grader (the tests that existed at the parent commit, plus the test added in the fix commit, if any).\n\nStep 1: mine five candidate tasks from git.\n\n```\n# Find bugfix-ish commits that also touched test files\nfor sha in $(git log --oneline -300 --grep='fix' -i --format='%h'); do\n  if git show --name-only --format='' $sha | grep -q 'test'; then\n    echo \"$sha $(git log -1 --format='%s' $sha)\"\n  fi\ndone | head -20\n```\n\nI pick five where the diff is under 60 lines. Small diffs keep the test honest — you're measuring one decision, not endurance.\n\nStep 2: for each chosen commit, build a task directory.\n\n``` bash\n#!/usr/bin/env bash\n# make_task.sh <fix_commit_sha> <task_name>\nset -euo pipefail\nSHA=$1; NAME=$2\nmkdir -p \"tasks/$NAME\"\ngit worktree add \"tasks/$NAME/repo\" \"${SHA}^\" 2>/dev/null\n# The 'gold' patch and the grading test come from the fix commit itself\ngit show \"$SHA\" -- . ':!*test*' > \"tasks/$NAME/gold.patch\"\ngit show \"$SHA\" -- '*test*' > \"tasks/$NAME/grading_test.patch\"\n# Prompt = commit message of the fix, i.e. the problem statement\ngit log -1 --format='%B' \"$SHA\" > \"tasks/$NAME/prompt.txt\"\n```\n\nNow each task has: a repo checked out *before* the fix, a problem statement (the original commit message), and a grader (the test that arrived with the fix). Nothing is hand-written.\n\nStep 3: run the model and grade.\n\n```\n# grade.py — run from tasks/<name>/repo after applying the model's patch\nimport subprocess, json, sys, pathlib\n\ndef run(cmd):\n    return subprocess.run(cmd, shell=True, capture_output=True, text=True)\n\nresult = {\"applies_clean\": False, \"tests_pass\": False, \"overreach\": False}\n\npatch = pathlib.Path(\"../model.patch\")\nif run(f\"git apply --check {patch}\").returncode == 0:\n    result[\"applies_clean\"] = True\n    run(f\"git apply {patch}\")\n    run(\"git apply ../grading_test.patch\")  # the hidden test\n    t = run(\"python -m pytest -x -q\")\n    result[\"tests_pass\"] = (t.returncode == 0)\n    # Overreach: did the model edit files outside the gold patch's scope?\n    gold_files = set(run(\"git apply --numstat ../gold.patch\").stdout.split())\n    changed = run(\"git diff --name-only\").stdout.split()\n    result[\"overreach\"] = len(set(changed) - set(gold_files)) > 1\n\nprint(json.dumps(result))\n```\n\nThree scores per task, five tasks, fifteen data points. That's it. I deliberately score `overreach`\n\nbecause in my experience the most expensive model failure isn't a wrong answer — it's a confident model quietly reformatting three unrelated files in the same patch.\n\nI ran this against two setups: my usual paid tool, and a free model served through MonkeyCode, which currently offers free model access plus a free server option for exactly this kind of side experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I picked it for the comparison run because the experiment's entire premise is \"is free good enough,\" and a zero-cost server meant I could rerun the harness as many times as I wanted without watching a meter. I'm not going to quote numbers as if they're universal — your repo is not my repo — but the *shape* of the result was consistent:\n\n| Task shape | Paid tool | Free model |\n|---|---|---|\n| One-line logic fix, existing failing test | pass | pass |\n| Off-by-one in pagination | pass | pass |\n| Refactor with hidden behavior requirement | pass | fail (missed edge case) |\n| SQL migration + backfill | pass | patch didn't apply cleanly |\n| Fix with ambiguous commit message | fail | fail |\n\nTwo honest observations. First, the free model was fine on the unambiguous tasks — which, embarrassingly, are most of my real tickets. Second, both setups failed the ambiguous-prompt task, which confirmed the harness is measuring task clarity as much as model quality. A task neither setup can solve is a bad task, not a bad model; I rewrote that prompt and both passed.\n\nUse this if you're evaluating coding assistants and you're tired of demo-driven decisions. Skip it if your work is greenfield-heavy — the whole method depends on having a fix-history to mine — or if you need statistically meaningful numbers for a procurement decision, in which case you need a bigger labeled set and more discipline than a bash script gives you.\n\nWhat changed for me: the question stopped being \"which model is best\" and became \"which tasks am I still doing by hand that a free model already handles.\" The harness took an afternoon; the answer paid for it in a week. If you want to replicate it, the three scripts above are the whole thing — point them at your own history, and if you need somewhere free to run the candidate model, that's a reasonable first use of MonkeyCode's free tier before you commit to anything.\n\nI'd be curious what task shapes other people's git histories turn up — mine were depressingly dominated by off-by-one errors.", "url": "https://wpnews.pro/news/vibes-are-not-a-benchmark-a-30-minute-harness-to-test-whether-a-free-coding-can", "canonical_source": "https://dev.to/hackrs_6393/vibes-are-not-a-benchmark-a-30-minute-harness-to-test-whether-a-free-coding-model-can-touch-your-3f94", "published_at": "2026-08-12 23:09:08+00:00", "updated_at": "2026-08-12 23:46:02.504526+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["MonkeyCode", "HumanEval"], "alternates": {"html": "https://wpnews.pro/news/vibes-are-not-a-benchmark-a-30-minute-harness-to-test-whether-a-free-coding-can", "markdown": "https://wpnews.pro/news/vibes-are-not-a-benchmark-a-30-minute-harness-to-test-whether-a-free-coding-can.md", "text": "https://wpnews.pro/news/vibes-are-not-a-benchmark-a-30-minute-harness-to-test-whether-a-free-coding-can.txt", "jsonld": "https://wpnews.pro/news/vibes-are-not-a-benchmark-a-30-minute-harness-to-test-whether-a-free-coding-can.jsonld"}}