{"slug": "pick-a-free-ai-model-by-score-not-by-reputation-a-20-prompt-harness", "title": "Pick a Free AI Model by Score, Not by Reputation: A 20-Prompt Harness", "summary": "A developer has created a 20-prompt evaluation harness for selecting AI models based on empirical scoring rather than reputation. The harness, designed for the open-source project MonkeyCode, runs real repository tasks against candidate models and scores outputs on a rubric, producing a decision table. The developer emphasizes that the ranking is specific to the prompts and should be re-run as workloads change.", "body_md": "Model selection is the new dependency pinning. You would not add a library to your lockfile because a blog post praised it; you would run its tests against your own code first. Most teams do the opposite with AI models: they pick one from a trending article, configure it once, and never re-score it. A 20-prompt harness turns that decision back into evidence.\n\nHere is the concrete situation I am working from. MonkeyCode is an open-source project whose free tier, at the time of writing, includes access to free models, a token allocation of 10 million, and a free server instance you can use for evaluation runs. Model lists and quotas move, so verify the current numbers in the docs before you depend on them.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nThe harness design matters more than the product behind it. You define 20 prompts from real tasks in your repository, send them to every model you are considering, and score the outputs with a rubric you can defend in a code review. The output is a decision table: model, prompt, pass, fail, and score. No opinions, no release-note reading.\n\nA generic prompt set measures generic skill. Your prompts should come from commits, issues, and failures that actually happened in your repo. Keep the list small but representative:\n\n```\n# eval/prompts.yaml\nprompts:\n  - id: commit-007\n    task: commit_message\n    input: fix webhook parser; handle empty body regression from PR 412\n  - id: test-013\n    task: generate_test\n    input: parseRetryAfter(value) returns null on absent header\n  - id: review-021\n    task: review_diff\n    input: review the unified diff in patches/pr-118.diff\n  - id: explain-034\n    task: explain_error\n    input: TypeError Cannot read properties of undefined reading map\n```\n\nTwenty prompts is a sample, not a census. It is enough to expose a stable ranking for one narrow task family, and small enough that a human can read every output in under an hour.\n\nKeep the runner provider-agnostic. The script below expects an endpoint and an API key from environment variables, so you can point it at any compatible model service, including the free models available through MonkeyCode's tier:\n\n``` bash\n#!/usr/bin/env bash\n# eval/run.sh - evaluate every model against every prompt and store raw output\nset -euo pipefail\nMODELS=\"${MODELS:-model-a,model-b}\"\nPROMPTS=\"eval/prompts.yaml\"\nOUTDIR=\"eval/results\"\nmkdir -p \"$OUTDIR\"\nmapfile -t ids < <(yq -r '.prompts[].id' \"$PROMPTS\")\n\nfor model in ${MODELS//,/ }; do\n  for id in \"${ids[@]}\"; do\n    prompt=$(yq -r \".prompts[] | select(.id == \\\"$id\\\") | .input\" \"$PROMPTS\")\n    ts=$(date +%s)\n    curl -sS \"$MODEL_ENDPOINT/v1/chat/completions\" \\\n      -H \"Authorization: Bearer $MODEL_API_KEY\" \\\n      -H \"Content-Type: application/json\" \\\n      -d '{\"model\":\"'\"$model\"'\",\"messages\":[{\"role\":\"user\",\"content\":\"'\"$prompt\"'\"}]}' \\\n      > \"$OUTDIR/${model}-${id}-${ts}.json\"\n    echo \"finished $model / $id\"\n  done\ndone\n```\n\nThis is a template, not a proven production tool: adjust the YAML parsing to your environment and test it against one model before running the full batch.\n\nHuman reading is the only honest scorer, so make the rubric mechanical enough to stay consistent. Score each output from 0 to 5 on four criteria:\n\nSum the four categories for a maximum of 20 points per prompt. A decision table, printed after the run, is the whole deliverable:\n\n| Model | Compiles | Fits bounds | Uses context | No invented API | Total / 100 |\n|---|---|---|---|---|---|\n| free-model-a | 4 | 3 | 4 | 2 | 65 |\n| free-model-b | 5 | 4 | 5 | 5 | 95 |\n| free-model-c | 3 | 5 | 2 | 4 | 70 |\n\nThe ranking belongs to your prompts, not to the model. Re-run the harness when your workload changes, because a commit-message winner is not necessarily a test-generation winner.\n\nThe second free resource is useful here. After the harness picks a winner, expose it through a small scheduled job on the free server instance: every night, generate outputs for ten new prompts and post the score to a channel. That server is an evaluation box, not a production host. Rate limits and cold starts make it a poor place for user-facing traffic, so treat it as a measurement device.\n\nA cron entry is enough:\n\n```\n0 3 * * * cd /srv/model-eval && ./run.sh && ./score.py > report.md\n```\n\nIf the score drops a full standard deviation below the baseline, the workflow should open an issue automatically. That alert is the whole point: models change silently, and your decision table goes stale without anyone noticing.\n\nImplementing this harness will touch real API costs. With a free tier the token budget is a constraint you should respect: 10 million tokens covers hundreds of evaluation runs, but a careless loop can burn it in an afternoon, so the runner records usage per call and fails early at 80 percent. The free server adds a real but modest compute ceiling; long batch runs may hit timeout walls, which is why the cron job is split per prompt rather than run as one giant request.\n\nNone of this validates correctness in a deep sense. A 95-point model can still generate a plausible but wrong test. The harness measures consistency, format fit, and contextual recall, not whether the logic matches the business requirements.\n\nNext time someone proposes a model by name in a planning meeting, do not argue about reputation. Ask which 20 prompts it passed. The free tier from MonkeyCode is a reasonable place to run the gauntlet, and the free server gives you a cheap way to keep the score fresh. Neither detail changes the method: score, compare, decide, then re-score when the prompt set changes.", "url": "https://wpnews.pro/news/pick-a-free-ai-model-by-score-not-by-reputation-a-20-prompt-harness", "canonical_source": "https://dev.to/codecpp_5026/pick-a-free-ai-model-by-score-not-by-reputation-a-20-prompt-harness-4gck", "published_at": "2026-08-30 11:20:45+00:00", "updated_at": "2026-08-30 11:52:36.566069+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "machine-learning"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/pick-a-free-ai-model-by-score-not-by-reputation-a-20-prompt-harness", "markdown": "https://wpnews.pro/news/pick-a-free-ai-model-by-score-not-by-reputation-a-20-prompt-harness.md", "text": "https://wpnews.pro/news/pick-a-free-ai-model-by-score-not-by-reputation-a-20-prompt-harness.txt", "jsonld": "https://wpnews.pro/news/pick-a-free-ai-model-by-score-not-by-reputation-a-20-prompt-harness.jsonld"}}