Model selection is the new dependency pinning. You would not add a library to your lockfile because a blog post praised it; you would run its tests against your own code first. Most teams do the opposite with AI models: they pick one from a trending article, configure it once, and never re-score it. A 20-prompt harness turns that decision back into evidence.
Here is the concrete situation I am working from. MonkeyCode is an open-source project whose free tier, at the time of writing, includes access to free models, a token allocation of 10 million, and a free server instance you can use for evaluation runs. Model lists and quotas move, so verify the current numbers in the docs before you depend on them.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The harness design matters more than the product behind it. You define 20 prompts from real tasks in your repository, send them to every model you are considering, and score the outputs with a rubric you can defend in a code review. The output is a decision table: model, prompt, pass, fail, and score. No opinions, no release-note reading.
A generic prompt set measures generic skill. Your prompts should come from commits, issues, and failures that actually happened in your repo. Keep the list small but representative:
prompts:
- id: commit-007
task: commit_message
input: fix webhook parser; handle empty body regression from PR 412
- id: test-013
task: generate_test
input: parseRetryAfter(value) returns null on absent header
- id: review-021
task: review_diff
input: review the unified diff in patches/pr-118.diff
- id: explain-034
task: explain_error
input: TypeError Cannot read properties of undefined reading map
Twenty prompts is a sample, not a census. It is enough to expose a stable ranking for one narrow task family, and small enough that a human can read every output in under an hour.
Keep the runner provider-agnostic. The script below expects an endpoint and an API key from environment variables, so you can point it at any compatible model service, including the free models available through MonkeyCode's tier:
#!/usr/bin/env bash
set -euo pipefail
MODELS="${MODELS:-model-a,model-b}"
PROMPTS="eval/prompts.yaml"
OUTDIR="eval/results"
mkdir -p "$OUTDIR"
mapfile -t ids < <(yq -r '.prompts[].id' "$PROMPTS")
for model in ${MODELS//,/ }; do
for id in "${ids[@]}"; do
prompt=$(yq -r ".prompts[] | select(.id == \"$id\") | .input" "$PROMPTS")
ts=$(date +%s)
curl -sS "$MODEL_ENDPOINT/v1/chat/completions" \
-H "Authorization: Bearer $MODEL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"'"$model"'","messages":[{"role":"user","content":"'"$prompt"'"}]}' \
> "$OUTDIR/${model}-${id}-${ts}.json"
echo "finished $model / $id"
done
done
This is a template, not a proven production tool: adjust the YAML parsing to your environment and test it against one model before running the full batch.
Human reading is the only honest scorer, so make the rubric mechanical enough to stay consistent. Score each output from 0 to 5 on four criteria:
Sum the four categories for a maximum of 20 points per prompt. A decision table, printed after the run, is the whole deliverable:
| Model | Compiles | Fits bounds | Uses context | No invented API | Total / 100 |
|---|---|---|---|---|---|
| free-model-a | 4 | 3 | 4 | 2 | 65 |
| free-model-b | 5 | 4 | 5 | 5 | 95 |
| free-model-c | 3 | 5 | 2 | 4 | 70 |
The ranking belongs to your prompts, not to the model. Re-run the harness when your workload changes, because a commit-message winner is not necessarily a test-generation winner.
The second free resource is useful here. After the harness picks a winner, expose it through a small scheduled job on the free server instance: every night, generate outputs for ten new prompts and post the score to a channel. That server is an evaluation box, not a production host. Rate limits and cold starts make it a poor place for user-facing traffic, so treat it as a measurement device.
A cron entry is enough:
0 3 * * * cd /srv/model-eval && ./run.sh && ./score.py > report.md
If the score drops a full standard deviation below the baseline, the workflow should open an issue automatically. That alert is the whole point: models change silently, and your decision table goes stale without anyone noticing.
Implementing this harness will touch real API costs. With a free tier the token budget is a constraint you should respect: 10 million tokens covers hundreds of evaluation runs, but a careless loop can burn it in an afternoon, so the runner records usage per call and fails early at 80 percent. The free server adds a real but modest compute ceiling; long batch runs may hit timeout walls, which is why the cron job is split per prompt rather than run as one giant request.
None of this validates correctness in a deep sense. A 95-point model can still generate a plausible but wrong test. The harness measures consistency, format fit, and contextual recall, not whether the logic matches the business requirements.
Next time someone proposes a model by name in a planning meeting, do not argue about reputation. Ask which 20 prompts it passed. The free tier from MonkeyCode is a reasonable place to run the gauntlet, and the free server gives you a cheap way to keep the score fresh. Neither detail changes the method: score, compare, decide, then re-score when the prompt set changes.