A Free Tier Benchmark Needs a Test Set, Not a Screenshot

MonkeyCode, an open-source coding project, has introduced a free tier offering ten million tokens and a free server option, and a developer has outlined a benchmarking method to evaluate such free AI coding tiers. The method uses a fixed task set of twenty unseen tasks from a real repository's commit history, with metrics including pass@k, median tokens per passing task, and failure logs, to provide repeatable and honest evaluation. The developer emphasizes that evaluation quality, not model quality, is often the variable being measured, and the free allocation makes the first benchmark run free as well.

A free token quota is a budget, and a budget without a burn-rate measurement is just a number in a marketing email. The only honest way to evaluate a free AI coding tier is to run it against a fixed task set with controlled variables and repeatable metrics. This article defines that method and provides a harness that produces numbers worth trusting. Recent DEV discussions about perfect harness scores and poor model scores point at the real problem, which is that evaluation quality, not model quality, is often the variable being measured. MonkeyCode, an open-source coding project, currently offers a free ten-million-token allocation and a free server option, and that offer is a useful test case for this method. A quota without a burn-rate measurement cannot be evaluated, and the free allocation makes the first benchmark run free as well. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A benchmark is only as honest as its task set, and the best task set is one the model has never seen. Pull twenty small tasks from a real repository's commit history, where each task contains a prompt, a setup command, a test command, and an output filename. The prompt describes the change, the setup installs fixtures, and the test asserts the behavior, so every output either passes or fails without human judgment. Here is the task format in JSONL, one task per line: {"id": "fix off by one", "prompt": "Fix the off-by-one error in range sum so that range sum 1, 5 returns 15.\n\ndef range sum start, end :\n total = 0\n for i in range start, end :\n total += i\n return total", "filename": "solution.py", "setup": "true", "test": "python -c \"from solution import range sum; assert range sum 1,5 ==15; assert range sum 2,2 ==2\"", "category": "bug fix"} {"id": "write fizzbuzz", "prompt": "Write a function fizzbuzz n that returns 'Fizz' for multiples of 3, 'Buzz' for multiples of 5, and 'FizzBuzz' for multiples of both.", "filename": "solution.py", "setup": "true", "test": "python -c \"from solution import fizzbuzz; assert fizzbuzz 3 =='Fizz'; assert fizzbuzz 5 =='Buzz'; assert fizzbuzz 15 =='FizzBuzz'\"", "category": "feature"} Three categories — bug fixes, features, and test writing — keep the set representative, and each task should take an experienced developer under ten minutes. Twenty tasks is the minimum viable sample because it gives a pass rate with a visible failure pattern instead of a single lucky or unlucky run. Sourcing tasks from a repository the team already maintains is the most honest option, because the commit messages describe the intended change and the test suite defines success. Public benchmark sets are convenient but risk contamination, since the model may have seen the exact prompts during training, and a small internal set of twenty tasks is worth more than a thousand public ones for this reason. The metrics that matter are pass@k, median tokens per passing task, median wall time, and the failure log. Run every task three times and report the median rather than the best, because the best run is the one the marketing page will quote. Token burn on failed attempts separates an engineering number from a marketing number, because a model that fails twice before succeeding costs three times the budget. The failure log matters more than the pass rate when the tier is free, since a free quota is consumed by failed attempts as well as successful ones, and a model that passes every task on the third try has a perfect pass rate and a terrible token economy. The harness must fix the model version, temperature, system prompt, and retry policy before the first run, then validate itself with two sanity checks. A known-broken task must fail and a known-good task must pass, because any contradiction means the harness, not the model, is what got benchmarked. Network variance is controlled by running from one region and recording latency per attempt, while a warm-up call absorbs cold-start effects before the first task. The sanity checks belong in the repository next to the task set, so a reviewer can rerun them before trusting any new result, and a harness that cannot fail on a broken task will eventually produce a perfect score for the wrong reason. The following Python script reads the task set, calls an OpenAI-compatible endpoint, writes each model output to the target file, runs the setup and test commands, and records every attempt: bash /usr/bin/env python3 """Free-tier benchmark harness: fixed tasks, controlled runs, honest numbers.""" import argparse import json import os import subprocess import tempfile import time from pathlib import Path from openai import OpenAI def load tasks path : lines = Path path .read text .splitlines return json.loads line for line in lines if line.strip def run shell script, cwd, timeout=60 : try: proc = subprocess.run script, shell=True, cwd=cwd, capture output=True, text=True, timeout=timeout, return proc.returncode == 0, proc.stdout -500: + proc.stderr -500: except subprocess.TimeoutExpired: return False, "timeout" def solve client, model, task : messages = {"role": "system", "content": "Write only the requested code. No explanations."}, {"role": "user", "content": task "prompt" }, started = time.monotonic reply = client.chat.completions.create model=model, messages=messages, temperature=0.2 elapsed = time.monotonic - started usage = reply.usage return reply.choices 0 .message.content, elapsed, usage.prompt tokens, usage.completion tokens def evaluate client, model, task, workdir : code, latency, prompt tokens, completion tokens = solve client, model, task task dir = Path workdir / task "id" task dir.mkdir parents=True, exist ok=True task dir / task.get "filename", "solution.py" .write text code or "" if task.get "setup" : ok, log = run shell task "setup" , task dir if not ok: return {"pass": False, "stage": "setup", "log": log} passed, log = run shell task "test" , task dir return { "pass": passed, "stage": "test", "latency s": round latency, 2 , "prompt tokens": prompt tokens, "completion tokens": completion tokens, "log": log, } def main : parser = argparse.ArgumentParser parser.add argument "--model", default=os.getenv "MODEL", "your-model-id" parser.add argument "--base-url", default=os.getenv "BASE URL" parser.add argument "--api-key", default=os.getenv "API KEY" parser.add argument "--tasks", type=Path, default=Path "tasks.jsonl" parser.add argument "--attempts", type=int, default=3 parser.add argument "--workdir", type=Path, default=Path tempfile.mkdtemp args = parser.parse args client = OpenAI base url=args.base url, api key=args.api key summary = {"model": args.model, "attempts": args.attempts, "tasks": } for task in load tasks args.tasks : attempts = evaluate client, args.model, task, args.workdir for in range args.attempts passed = sum 1 for attempt in attempts if attempt "pass" tokens = sorted a "prompt tokens" + a "completion tokens" for a in attempts latency = sorted a "latency s" for a in attempts summary "tasks" .append { "id": task "id" , "pass at k": passed / args.attempts, "median tokens": tokens len tokens // 2 , "median latency s": latency len latency // 2 , } print json.dumps summary "tasks" -1 rate = sum t "pass at k" for t in summary "tasks" / len summary "tasks" total = sum t "median tokens" for t in summary "tasks" print json.dumps {"pass rate": round rate, 3 , "total median tokens": total} if name == " main ": main Run it against the endpoint of the tier you are evaluating: export BASE URL="https://api.example.com/v1" export API KEY="your-key" export MODEL="your-model-id" python3 benchmark.py --attempts 3 The script prints one JSON line per task and a final summary with the pass rate and median token burn, which is the number the quota announcement never mentions. A pass rate without a failure log is a claim, and a token count without a task set is a guess, so the report must include both. The output shows whether a tier passes twelve of twenty tasks with two hundred thousand median tokens or eighteen of twenty with eight hundred thousand. Those two profiles demand opposite engineering decisions, and the quota announcement cannot tell them apart. A ten-million-token allocation changes the calculation only when the burn rate is known, because a tier that burns three times the expected tokens on failed attempts consumes the quota three times faster. This benchmark measures well-scoped tasks that a developer can verify in minutes, so it says nothing about long-horizon agentic work, repo-scale refactors, or subjective code quality. Teams that need production guarantees should not route critical paths through a free tier on a twenty-task pass rate, and vendor comparisons require an identical task set to mean anything. The method is a first filter for whether a free tier deserves real project time, not a certification of production readiness. Running this harness against MonkeyCode's free model access and free server option takes an afternoon, and the resulting JSON says more than the quota announcement ever will.