Zero Cost Is Not Zero Variance: Four Myths About Repeating an AI Coding Run

A developer published a myth-busting guide arguing that free AI coding tiers should be judged by run-to-run variance rather than price alone, since identical prompts against the same commit rarely produce identical diffs. The writeup recommends a repeatable ledger that records exit codes, test outcomes, diff counts and wall-clock time across roughly eight runs, then derives a tail ratio (worst run divided by median run) to distinguish tiers suitable for exploration from those safe for deadline work. It includes a bash harness that resets the worktree between attempts and appends one CSV row per run.

Imagine the same prompt, the same repository, and the same command, run on two different afternoons. On the first afternoon the agent finishes quickly and the test suite is green; on the second it edits four extra files and leaves two tests red. Nothing in the repository changed between those runs, and that is precisely why the second outcome is difficult to defend in a code review. Most comparisons of free AI coding tiers begin and end with price. The question that actually changes your workflow is whether a tier behaves consistently enough that you can build a habit around it, because a cheap tool with a wide outcome spread quietly spends your review time instead. This article converts that spread into a small ledger you can run in one sitting, and it retires four claims that developers repeat whenever tiers are compared. The format is a myth-busting FAQ, because myths are what get repeated in chat threads while the measurements stay private. Each claim below comes with a cheap check and a corrected mental model, and none of the checks requires a paid plan to be interesting. Repeat a task eight times against the same commit and you will rarely get eight identical diffs. Sampling settings, tool-call ordering, network retries, and the order in which files are read all push the run in slightly different directions. A single green run is therefore an anecdote about one sample, not a property of the tier. The corrected mental model is that you are not buying a function, you are sampling a distribution. That sounds abstract until you write the distribution down, and writing it down takes less effort than arguing about it in a pull request. Once you have rows, the conversation moves from "it worked for me" to "median acceptable, tail unacceptable". A passing test suite proves that one sample passed, and it says nothing about how often the next sample will pass. The failure mode is subtle: the agent's diff and the test outcome are recorded together in your head, but only the happy version survives into the team's memory. Recording both columns for every run is the entire fix. The corrected model separates process from task. Exit code zero tells you the command finished; a test result tells you the change was acceptable; a diff count tells you how much review surface the change created. Those three signals disagree often enough that collapsing them into one boolean will mislead you. Medians are comfortable and tails are expensive, and free tiers tend to produce tails. If seven runs finish in a few minutes and the eighth takes an hour, your median looks healthy while your afternoon is gone. Anyone who has waited on a stuck job knows the average is not the number that ruins the day. The corrected model is a tail ratio: worst run divided by median run, computed from the same ledger rows. A tier with a stable median and a large tail ratio is fine for exploration and hostile for anything with a deadline. Compute both numbers before you decide which kind of tool you are holding. The harness below is deliberately boring, and you should adapt the one line that invokes your agent. It records the commit you started from, resets the worktree between runs, and appends one CSV row per attempt. Treat it as a proposal to run on your own task rather than as a benchmark result, since the numbers you get will depend on your repository. bash /usr/bin/env bash variance ledger.sh — repeat one task N times, record one row per run set -uo pipefail N="${N:-8}" TASK="${TASK:-tasks/fix import order.md}" OUT="ledger/$ date -u +%Y%m%dT%H%M%SZ " mkdir -p "$OUT" git rev-parse HEAD "$OUT/base commit.txt" git status --porcelain "$OUT/base dirty.txt" should be empty for i in $ seq 1 "$N" ; do start=$ date +%s Replace this line with your agent's non-interactive entry point. agent run --task "$TASK" --no-interactive "$OUT/run $i.log" 2 &1 code=$? end=$ date +%s added deleted=$ git diff --numstat | awk '{a+=$1; d+=$2} END {print a+0 " " d+0 }' printf '%s,%s,%s,%s\n' "$i" "$ end-start " "$code" "$added deleted" "$OUT/ledger.csv" git checkout -- . /dev/null 2 &1 git clean -fdq done The reset step is what makes the rows comparable, and skipping it is the most common way to poison a ledger. Running without a clean worktree means run two starts from run one's leftovers, and the whole table then measures compounding edits instead of tier behaviour. Check base dirty.txt before you trust anything downstream. Summarising takes a short awk script, because pulling in a dataframe library for eight rows is unnecessary ceremony. awk -F, '{wall NR =$2; n=NR} END{ for i=1;i<=n;i++ for j=i+1;j<=n;j++ if wall j <wall i {t=wall i ;wall i =wall j ;wall j =t} med= n%2 ?wall n+1 /2 : wall n/2 +wall n/2+1 /2 printf "runs=%d min=%ss median=%ss max=%ss tail ratio=%.2f\n", n, wall 1 , med, wall n , wall n /med }' ledger/ /ledger.csv Add the task verdict yourself, because wall time cannot tell you whether a diff was correct. A second column holding the exit status of your test command is usually enough, and it keeps the ledger honest when a fast run produces a bad patch. Variance usually arrives from the environment wrapped around the model. A shared shell with a warm package cache, a locale that formats numbers differently, and a proxy that silently retries a request are three common sources that have nothing to do with the weights. If you change the environment between runs, you have measured the environment. The corrected model treats the agent as one variable among several, and it forces you to hold the others fixed or record them. A warm cache and a different locale are environment variables in the literal sense, and both explain why two teams can disagree about the same tier while both are reporting honestly. Repetition is the step everyone skips, because repetition is what costs money and occupies a machine. MonkeyCode's free model access and free server option change the arithmetic of that step, since an eight-run ledger stops being an expense and becomes an ordinary afternoon task. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The honest version of that claim is narrower than a slogan. Free access makes sampling practical at small n, and a free server keeps the loop off your laptop while you review the previous run; neither makes a tier deterministic. Check the project's own documentation for current allowances and machine characteristics, because quotas and hardware are exactly the details that go stale fastest. Limitations worth stating plainly: eight runs is a small sample, diff size is not correctness, wall time includes network noise you cannot control, and every repetition consumes allowance you may want later. The ledger also assumes a non-interactive entry point, which not every agent exposes cleanly. Treat the output as a screen for whether a tier deserves careful measurement, not as a benchmark. Teams that need deterministic output, contractual latency, or regulated audit trails will not be served by sampling a free tier at all. Anyone whose task takes an hour per run should shrink the task before running any ledger. And if review capacity is the real bottleneck, variance reduction matters more than cost reduction, so pay for stability rather than collecting cheap rows. The myth-busting habit generalises beyond any single product: write the claim down, find the cheapest observation that could refute it, and keep the rows. Eight runs and four columns will not settle a platform debate, but they will stop you from repeating a one-sample story as if it were a property. If you want to try the ledger, the free tier is enough to learn whether your own task is stable before you commit real budget to it.