Your eval score moved from 0.81 to 0.84. You shipped the change. Next week the old baseline won again.
Was that improvement real? Or was it noise wearing a decimal point?
I keep meeting the same pattern in agent work. A single number gets treated as a measurement. It is usually an anecdote with a decimal.
Here are six myths I keep running into, plus a small harness that reports the number people actually need: how often an item flips.
Hosted inference is not a function. It is a service with batching, sampling, and version drift.
Temperature zero reduces variance. It does not delete variance. Batching and backend changes still move tokens around.
The corrected mental model: every call is a sample. One call is a sample of size one.
So what does that make your score? A sample mean of an unknown distribution.
Means hide flips. Two runs can both score 0.80 and disagree on completely different items.
Ask a sharper question instead. How many items changed verdict between runs?
That number is a flip rate. It tells you whether your delta is signal or jitter.
A mean answers "how did we do?" A flip rate answers "can I trust the diff?" You need the second one to ship.
If an LLM grades your outputs, the grader is also a sampled system. Rubric wording moves it. Item order moves it. Context length moves it.
So measure the judge too. Feed the same saved outputs through it twice. If the judge flips, your score is a coin toss with extra steps.
Prefer exact checks wherever possible:
jsonschema for structured output.
Keep the LLM judge for the cases nothing else can grade. Then report judge agreement next to the score.
Prompt caching speeds things up. It also changes what the model actually sees.
Deduplication shrinks your item set. Your denominator changes. Your score moves for free.
Record the item count and a hash of the input file in every report. If the hash changed, the comparison is not a comparison.
Files get edited. Items get added. Ordering changes. Someone quietly fixes a gold label.
Pin the dataset by hash, not by path. items.jsonl is a location. A SHA-256 is an identity.
When a score jumps, check the dataset hash first. Then check the prompt hash. Then argue about the model.
A 0.03 delta inside a 0.12 noise band is decoration, not progress.
Set the gate from observed variance, not from hope. Run the harness a few times before you trust any threshold.
And be honest about what you are comparing. A new prompt against an old score is a diff of two different weeks, not two different prompts.
The script below runs each item several times and reports per-item flip rate. It is illustrative, not a benchmark. Point it at your own items and endpoint.
import hashlib, json, os, statistics, urllib.request
BASE = os.environ['MODEL_BASE_URL'].rstrip('/')
KEY = os.environ['MODEL_API_KEY']
MODEL = os.environ['MODEL_NAME']
RUNS = int(os.environ.get('RUNS', '5'))
TEMP = float(os.environ.get('TEMPERATURE', '0'))
def sha(text: str) -> str:
return hashlib.sha256(text.encode()).hexdigest()[:12]
def ask(item, seed):
body = json.dumps({
'model': MODEL,
'messages': item['messages'],
'temperature': TEMP,
'seed': seed,
}).encode()
req = urllib.request.Request(
BASE + '/chat/completions',
data=body,
headers={'Authorization': 'Bearer ' + KEY,
'Content-Type': 'application/json'})
with urllib.request.urlopen(req, timeout=120) as r:
return json.load(r)['choices'][0]['message']['content']
def normalize(text: str) -> str:
return ' '.join(text.strip().lower().split())
def flip_rate(outputs):
counts = {}
for o in outputs:
key = normalize(o)
counts[key] = counts.get(key, 0) + 1
return 1 - max(counts.values()) / len(outputs)
items = [json.loads(line) for line in open('items.jsonl')]
report, worst = [], 0.0
for item in items:
outs = [ask(item, seed=1000 + i) for i in range(RUNS)]
fr = flip_rate(outs)
worst = max(worst, fr)
report.append({
'id': item['id'],
'flip_rate': round(fr, 2),
'distinct_outputs': len({normalize(o) for o in outs}),
'sample_hash': sha(outs[0]),
})
manifest = {
'model': MODEL,
'temperature': TEMP,
'runs': RUNS,
'items_hash': sha(open('items.jsonl').read()),
'mean_flip_rate': round(
statistics.mean(r['flip_rate'] for r in report), 3),
}
print(json.dumps({'manifest': manifest, 'items': report}, indent=2))
raise SystemExit(1 if worst > float(os.environ.get('MAX_FLIP', '0.2')) else 0)
Run it like this in CI:
pip install --quiet jsonschema
RUNS=5 MAX_FLIP=0.2 python flip_rate.py > eval_report.json
Two notes on the code. seed is a hint, not a contract; providers treat it differently, so verify in your own docs. And the script exits non-zero on a bad item, which makes it a gate instead of a dashboard nobody reads.
| Flip rate | What it suggests | Reasonable action |
|---|---|---|
| 0.00 | Stable on this item set | Trust deltas above your noise floor |
| 0.01 - 0.10 | Mild jitter | Require larger deltas before shipping |
| 0.11 - 0.30 | Unstable items | Fix grading or tighten the prompt |
| Above 0.30 | Coin toss | Stop tuning and repair the harness |
These bands are a starting heuristic, not measured constants. Calibrate them on your own set before you enforce anything.
This harness is repetitive by design. It wants several calls per item and a place to run them.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The operator describes free model access and a free server option. Those two things are genuinely useful here, because flip-rate measurement burns calls and wall-clock time for no user-visible feature.
A free endpoint lets you take the first pass across more items. A free server lets the loop run somewhere other than your laptop lid. I cannot speak to quotas, limits, or longevity, and you should check current terms before you build a pipeline on it.
Point the harness at whatever endpoint you have. The script does not care who answers.
Be clear about what this does not do.
429, stop and back off. Do not wrap the harness in a hot retry loop.
Skip this approach if you have fewer than about ten items. Skip it for smoke tests where any answer is acceptable. Skip it for one-shot creative tasks with no ground truth to compare.
Use it when a prompt change needs a decision, and the only evidence is a number that keeps wobbling.
Before you cite the next score, ask one thing. If I re-ran this today, how many items would change?
If you cannot answer that, you have a decimal point, not a measurement.