# A Free Tier Benchmark Needs a Test Set, Not a Screenshot

> Source: <https://dev.to/apppro_5726/a-free-tier-benchmark-needs-a-test-set-not-a-screenshot-jb7>
> Published: 2026-08-28 04:07:10+00:00

A free token quota is a budget, and a budget without a burn-rate measurement is just a number in a marketing email. The only honest way to evaluate a free AI coding tier is to run it against a fixed task set with controlled variables and repeatable metrics. This article defines that method and provides a harness that produces numbers worth trusting.

Recent DEV discussions about perfect harness scores and poor model scores point at the real problem, which is that evaluation quality, not model quality, is often the variable being measured. MonkeyCode, an open-source coding project, currently offers a free ten-million-token allocation and a free server option, and that offer is a useful test case for this method. A quota without a burn-rate measurement cannot be evaluated, and the free allocation makes the first benchmark run free as well.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A benchmark is only as honest as its task set, and the best task set is one the model has never seen. Pull twenty small tasks from a real repository's commit history, where each task contains a prompt, a setup command, a test command, and an output filename. The prompt describes the change, the setup installs fixtures, and the test asserts the behavior, so every output either passes or fails without human judgment.

Here is the task format in JSONL, one task per line:

```
{"id": "fix_off_by_one", "prompt": "Fix the off-by-one error in range_sum so that range_sum(1, 5) returns 15.\n\ndef range_sum(start, end):\n    total = 0\n    for i in range(start, end):\n        total += i\n    return total", "filename": "solution.py", "setup": "true", "test": "python -c \"from solution import range_sum; assert range_sum(1,5)==15; assert range_sum(2,2)==2\"", "category": "bug_fix"}
{"id": "write_fizzbuzz", "prompt": "Write a function fizzbuzz(n) that returns 'Fizz' for multiples of 3, 'Buzz' for multiples of 5, and 'FizzBuzz' for multiples of both.", "filename": "solution.py", "setup": "true", "test": "python -c \"from solution import fizzbuzz; assert fizzbuzz(3)=='Fizz'; assert fizzbuzz(5)=='Buzz'; assert fizzbuzz(15)=='FizzBuzz'\"", "category": "feature"}
```

Three categories — bug fixes, features, and test writing — keep the set representative, and each task should take an experienced developer under ten minutes. Twenty tasks is the minimum viable sample because it gives a pass rate with a visible failure pattern instead of a single lucky or unlucky run. Sourcing tasks from a repository the team already maintains is the most honest option, because the commit messages describe the intended change and the test suite defines success. Public benchmark sets are convenient but risk contamination, since the model may have seen the exact prompts during training, and a small internal set of twenty tasks is worth more than a thousand public ones for this reason.

The metrics that matter are pass@k, median tokens per passing task, median wall time, and the failure log. Run every task three times and report the median rather than the best, because the best run is the one the marketing page will quote. Token burn on failed attempts separates an engineering number from a marketing number, because a model that fails twice before succeeding costs three times the budget. The failure log matters more than the pass rate when the tier is free, since a free quota is consumed by failed attempts as well as successful ones, and a model that passes every task on the third try has a perfect pass rate and a terrible token economy.

The harness must fix the model version, temperature, system prompt, and retry policy before the first run, then validate itself with two sanity checks. A known-broken task must fail and a known-good task must pass, because any contradiction means the harness, not the model, is what got benchmarked. Network variance is controlled by running from one region and recording latency per attempt, while a warm-up call absorbs cold-start effects before the first task. The sanity checks belong in the repository next to the task set, so a reviewer can rerun them before trusting any new result, and a harness that cannot fail on a broken task will eventually produce a perfect score for the wrong reason.

The following Python script reads the task set, calls an OpenAI-compatible endpoint, writes each model output to the target file, runs the setup and test commands, and records every attempt:

``` bash
#!/usr/bin/env python3
"""Free-tier benchmark harness: fixed tasks, controlled runs, honest numbers."""
import argparse
import json
import os
import subprocess
import tempfile
import time
from pathlib import Path

from openai import OpenAI

def load_tasks(path):
    lines = Path(path).read_text().splitlines()
    return [json.loads(line) for line in lines if line.strip()]

def run_shell(script, cwd, timeout=60):
    try:
        proc = subprocess.run(
            script, shell=True, cwd=cwd,
            capture_output=True, text=True, timeout=timeout,
        )
        return proc.returncode == 0, proc.stdout[-500:] + proc.stderr[-500:]
    except subprocess.TimeoutExpired:
        return False, "timeout"

def solve(client, model, task):
    messages = [
        {"role": "system", "content": "Write only the requested code. No explanations."},
        {"role": "user", "content": task["prompt"]},
    ]
    started = time.monotonic()
    reply = client.chat.completions.create(model=model, messages=messages, temperature=0.2)
    elapsed = time.monotonic() - started
    usage = reply.usage
    return reply.choices[0].message.content, elapsed, usage.prompt_tokens, usage.completion_tokens

def evaluate(client, model, task, workdir):
    code, latency, prompt_tokens, completion_tokens = solve(client, model, task)
    task_dir = Path(workdir) / task["id"]
    task_dir.mkdir(parents=True, exist_ok=True)
    (task_dir / task.get("filename", "solution.py")).write_text(code or "")
    if task.get("setup"):
        ok, log = run_shell(task["setup"], task_dir)
        if not ok:
            return {"pass": False, "stage": "setup", "log": log}
    passed, log = run_shell(task["test"], task_dir)
    return {
        "pass": passed,
        "stage": "test",
        "latency_s": round(latency, 2),
        "prompt_tokens": prompt_tokens,
        "completion_tokens": completion_tokens,
        "log": log,
    }

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", default=os.getenv("MODEL", "your-model-id"))
    parser.add_argument("--base-url", default=os.getenv("BASE_URL"))
    parser.add_argument("--api-key", default=os.getenv("API_KEY"))
    parser.add_argument("--tasks", type=Path, default=Path("tasks.jsonl"))
    parser.add_argument("--attempts", type=int, default=3)
    parser.add_argument("--workdir", type=Path, default=Path(tempfile.mkdtemp()))
    args = parser.parse_args()

    client = OpenAI(base_url=args.base_url, api_key=args.api_key)
    summary = {"model": args.model, "attempts": args.attempts, "tasks": []}

    for task in load_tasks(args.tasks):
        attempts = [evaluate(client, args.model, task, args.workdir) for _ in range(args.attempts)]
        passed = sum(1 for attempt in attempts if attempt["pass"])
        tokens = sorted(a["prompt_tokens"] + a["completion_tokens"] for a in attempts)
        latency = sorted(a["latency_s"] for a in attempts)
        summary["tasks"].append({
            "id": task["id"],
            "pass_at_k": passed / args.attempts,
            "median_tokens": tokens[len(tokens) // 2],
            "median_latency_s": latency[len(latency) // 2],
        })
        print(json.dumps(summary["tasks"][-1]))

    rate = sum(t["pass_at_k"] for t in summary["tasks"]) / len(summary["tasks"])
    total = sum(t["median_tokens"] for t in summary["tasks"])
    print(json.dumps({"pass_rate": round(rate, 3), "total_median_tokens": total}))

if __name__ == "__main__":
    main()
```

Run it against the endpoint of the tier you are evaluating:

```
export BASE_URL="https://api.example.com/v1"
export API_KEY="your-key"
export MODEL="your-model-id"
python3 benchmark.py --attempts 3
```

The script prints one JSON line per task and a final summary with the pass rate and median token burn, which is the number the quota announcement never mentions.

A pass rate without a failure log is a claim, and a token count without a task set is a guess, so the report must include both. The output shows whether a tier passes twelve of twenty tasks with two hundred thousand median tokens or eighteen of twenty with eight hundred thousand. Those two profiles demand opposite engineering decisions, and the quota announcement cannot tell them apart. A ten-million-token allocation changes the calculation only when the burn rate is known, because a tier that burns three times the expected tokens on failed attempts consumes the quota three times faster.

This benchmark measures well-scoped tasks that a developer can verify in minutes, so it says nothing about long-horizon agentic work, repo-scale refactors, or subjective code quality. Teams that need production guarantees should not route critical paths through a free tier on a twenty-task pass rate, and vendor comparisons require an identical task set to mean anything. The method is a first filter for whether a free tier deserves real project time, not a certification of production readiness.

Running this harness against MonkeyCode's free model access and free server option takes an afternoon, and the resulting JSON says more than the quota announcement ever will.