# A Sandbox-First Workflow for Evaluating AI Coding Models on a Zero Budget

> Source: <https://dev.to/hackjs_7468/a-sandbox-first-workflow-for-evaluating-ai-coding-models-on-a-zero-budget-2kh7>
> Published: 2026-08-10 08:15:51+00:00

There's a conversation happening right now about what happens when we hand AI agents more tools and the boundaries fail. It's a good conversation, but it skips a step most of us hit first: before you worry about an agent escaping its sandbox, you have to pick a model, wire it into a workflow, and figure out whether it actually helps — ideally without putting a credit card behind an experiment that might go nowhere.

This article is about that earlier step. It's a repeatable workflow I've structured for evaluating AI coding assistance on side projects where the budget is literally zero, using a fixed prompt suite, a throwaway git repo, and free-tier tooling. The workflow doesn't depend on any single provider, but I'll show where free model access and a hosted free server slot fit naturally, because that combination removes the two most common blockers: API cost anxiety and "my laptop can't run this locally."

Most developers evaluate AI coding tools the way they evaluate a new keyboard — vibes. You paste one prompt, the output looks plausible, and you either adopt the tool or dismiss it based on a sample size of one. That's evaluation debt, and it compounds: you end up trusting a model on tasks it's bad at, or abandoning one that would have saved you hours on the tasks it's good at.

The fix is boring: treat model evaluation like a benchmark you can rerun, not a first impression.

The whole workflow lives in a disposable git repo. Nothing here touches production code, real secrets, or private repositories.

**Step 1 — Build a fixed prompt suite.** Pick 5–8 tasks that represent *your* actual work. Mine tend to cluster into four categories:

| Task type | Example prompt | What it reveals |
|---|---|---|
| Greenfield generation | "Write a rate limiter middleware for Express with sliding-window logic" | Can it produce runnable code, not just plausible code? |
| Bug localization | Paste a failing test + source file, ask for the root cause | Does it reason about existing code or hallucinate fixes? |
| Refactor with constraints | "Extract this into a pure function; no new dependencies" | Does it respect constraints or ignore half of them? |
| Explanation | "Explain what this regex does and where it backtracks" | Is it useful for onboarding/reading, not just writing? |

Keep the prompts in a file, version them, and never tune them to flatter a specific model.

**Step 2 — Run each prompt through a harness that captures everything.** Here's a minimal one. It's a runnable starting point, not a finished product:

``` bash
#!/usr/bin/env python3
"""eval_harness.py — run a prompt suite against an OpenAI-compatible endpoint
and log raw responses for offline review."""
import json, time, urllib.request, pathlib, sys

ENDPOINT = sys.argv[1]            # e.g. your free server's /v1/chat/completions URL
MODEL    = sys.argv[2]
SUITE    = pathlib.Path("prompt_suite.jsonl")  # one {"id": ..., "prompt": ...} per line
OUT      = pathlib.Path("results") / f"{MODEL}-{int(time.time())}.jsonl"
OUT.parent.mkdir(exist_ok=True)

for line in SUITE.read_text().splitlines():
    case = json.loads(line)
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": case["prompt"]}],
        "temperature": 0
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Content-Type": "application/json"})
    t0 = time.time()
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            resp = json.loads(r.read())
        text = resp["choices"][0]["message"]["content"]
    except Exception as e:
        text = f"__ERROR__: {e}"
    OUT.open("a").write(json.dumps({
        "id": case["id"], "model": MODEL,
        "latency_s": round(time.time() - t0, 2), "response": text
    }) + "\n")
    print(f"{case['id']}: done")
```

Deliberate choices: temperature 0 for repeatability, raw responses saved verbatim, errors recorded instead of retried away. Latency is logged but I treat it as a smoke signal, not a benchmark — free tiers throttle, and that's fine.

**Step 3 — Score outputs against acceptance criteria you wrote before seeing the results.** For code-generation prompts, the criterion is mechanical: does it run? For the rate-limiter example, that means literally dropping the output into the sandbox repo and running a pre-written test file. For bug localization, the criterion is whether the identified root cause matches the one you planted. Write the tests first; otherwise you'll grade leniently.

**Step 4 — Record a one-line verdict per task type.** After two or three runs, patterns emerge fast. In my experience structuring suites like this, models tend to have sharp edges — strong at greenfield generation, weak at constraint-heavy refactors, or vice versa — and the verdict table is what turns "this model feels mid" into "use it for scaffolding, don't trust it for surgical edits."

The workflow above assumes an OpenAI-compatible HTTP endpoint, which is the common denominator across providers. The friction is usually getting one without a billing account.

*Disclosure: This article was prepared as part of MonkeyCode's product outreach.*

MonkeyCode currently offers free model access and a free server option, which maps onto this workflow in a specific way: the free server gives you the endpoint for `eval_harness.py`

without provisioning anything, and the free model access means the suite can run to completion without you watching a meter. That's genuinely useful for the *evaluation* phase specifically, because evaluation is where cost anxiety does the most damage — people cut their prompt suite short, which is exactly how you end up back at vibes-based adoption.

One honest caveat: I can't tell you which models, quotas, or how long the free tier lasts, because those change and you should check the current terms before building a habit on them. Design your harness so the endpoint is a command-line argument — as in the script above — and swapping providers later is a one-line change. Never hardcode a free tier into your process.

A versioned prompt suite, a throwaway repo, and a 40-line harness turn "is this model any good" from a vibe into a verdict you can rerun next month when the model landscape shifts again — which it will. Free access tiers are best used exactly here: lowering the cost of being rigorous *before* you commit, not after.

If you've built your own evaluation suite, I'm curious which task categories exposed the biggest gaps between models — that's the data point I find hardest to get from public benchmarks.