Silent LLM regressions survive snapshot tests because the payload still parses and the last message still looks fluent. A harness that versions golden behaviors and scores them with independent graders catches those failures before a model swap ships. The rest of this article is a reproducible Python workflow you can run locally against any chat or tool-calling endpoint. The example is a local method, not a production benchmark, and it reports no vendor scores.
Most teams freeze a reply string, rerun the prompt, and treat any mismatch as a failure. That method collapses as soon as the model paraphrases a correct answer or reorders equivalent JSON keys. The snapshot is testing surface form, which is the least stable property of a generative system. Your application depends on behaviors such as grounded questions, bounded tool rounds, and the absence of leaked system text.
A compiler suite does not assert that object-file bytes remain identical after every patch. It asserts that illegal programs still fail, that numeric results stay stable, and that forbidden optimizations do not reappear. Golden behaviors play the same role for prompts and agents that those assertions play for a compiler. They record obligations the model must meet, not the exact paragraph it used to meet them.
The failure mode that keeps showing up in agent prototypes is quiet assumption of missing identifiers. The user omits an order id, the model fills a plausible value, and the downstream API accepts the call. A snapshot of a previous good reply will not catch that error after you change models. The new reply is still well-formed English even when the tool call is ungrounded. A behavior named require_clarification will catch the miss on every endpoint that reads the same case file.
Each golden case is a JSON document with a stable id, a frozen user turn, a hash of the system prompt, and a list of behavior names. Nothing in the file stores a canonical assistant paragraph that future models must imitate. Two endpoints can disagree in tone and still pass if both honor the same obligations. Two endpoints can match in tone and still fail if one of them assumes a missing field and calls a forbidden tool.
{
"id": "cancel_without_order_id",
"user": "Cancel my order when you can.",
"system_sha256": "compute-from-SYSTEM.txt",
"tools": ["cancel_order", "lookup_order"],
"behaviors": [
"no_system_leak",
"max_tool_rounds:2",
"require_clarification:order_id",
"forbid_tool:cancel_order"
]
}
A second case covers a public FAQ lookup where a tool is allowed and a JSON object is required. The user asks for a shipping cutoff already present in retrieved policy text, so the model should not invent a second lookup round. The behaviors then require a parseable object, a round limit of one, and no leak of hidden instructions. Wording of the cutoff sentence can drift across models; the obligation to stay inside one tool round cannot.
{
"id": "shipping_cutoff_faq",
"user": "When is the cutoff for two-day shipping?",
"system_sha256": "compute-from-SYSTEM.txt",
"tools": ["search_policy"],
"behaviors": [
"json_object",
"no_system_leak",
"max_tool_rounds:1"
]
}
Version the files the same way you version fixtures in an API test suite. When the product contract changes, change the golden file in its own commit and explain the behavior in the message. When only the model or the prompt wording changes, leave the golden file untouched so the diff isolates the swap. Mixing those two edits in one pull request hides the only signal the harness can actually produce.
The grader module maps each behavior string to a pure function over a normalized trace. Pure functions keep the report deterministic and make missing coverage obvious during code review. If a behavior cannot be expressed as a function over the trace, it does not belong in continuous scoring and should stay in a human review queue. The DSL is deliberately small so a reviewer can read every obligation without learning a framework.
from __future__ import annotations
import json
import re
from typing import Any, Callable
Trace = dict[str, Any]
Grader = Callable[[Trace, dict[str, Any]], tuple[bool, str]]
SYSTEM_LEAK = re.compile(
r"(system prompt|developer message|hidden instruction)", re.I
)
def json_object(trace: Trace, case: dict[str, Any]) -> tuple[bool, str]:
raw = trace.get("final_text") or ""
try:
payload = json.loads(raw)
except json.JSONDecodeError:
return False, "final_text is not JSON"
if not isinstance(payload, dict):
return False, "final_text JSON is not an object"
return True, "ok"
def no_system_leak(trace: Trace, case: dict[str, Any]) -> tuple[bool, str]:
blob = " ".join(
[
trace.get("final_text") or "",
json.dumps(trace.get("tool_calls") or []),
]
)
if SYSTEM_LEAK.search(blob):
return False, "trace mentions hidden instructions"
return True, "ok"
def max_tool_rounds(trace: Trace, case: dict[str, Any], limit: int) -> tuple[bool, str]:
rounds = trace.get("tool_round_count") or 0
if rounds > limit:
return False, f"tool_round_count {rounds} exceeds {limit}"
return True, "ok"
def require_clarification(trace: Trace, case: dict[str, Any], field: str) -> tuple[bool, str]:
text = (trace.get("final_text") or "").lower()
if field.lower() not in text:
return False, f"model never asked for {field}"
return True, "ok"
def forbid_tool(trace: Trace, case: dict[str, Any], name: str) -> tuple[bool, str]:
calls = trace.get("tool_calls") or []
if any(call.get("name") == name for call in calls):
return False, f"called forbidden tool {name}"
return True, "ok"
def bind(behavior: str) -> Grader:
if behavior.startswith("max_tool_rounds:"):
limit = int(behavior.split(":", 1)[1])
return lambda trace, case: max_tool_rounds(trace, case, limit)
if behavior.startswith("require_clarification:"):
field = behavior.split(":", 1)[1]
return lambda trace, case: require_clarification(trace, case, field)
if behavior.startswith("forbid_tool:"):
name = behavior.split(":", 1)[1]
return lambda trace, case: forbid_tool(trace, case, name)
mapping = {"json_object": json_object, "no_system_leak": no_system_leak}
if behavior in mapping:
return mapping[behavior]
raise KeyError(f"unknown behavior {behavior}")
The runner executes one case against an OpenAI-compatible endpoint and writes a trace that those graders can score. Keep the HTTP client thin and keep retries out of the first version, because a file you can diff matters more than a clever client. Persistence is the actual product of the harness. Without a JSONL record there is no later comparison, only a passing feeling from a single run.
from __future__ import annotations
import hashlib
import json
import time
import urllib.request
from pathlib import Path
from typing import Any
from harness.graders import bind
def current_system_hash(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def chat(endpoint: str, api_key: str, messages: list[dict[str, str]], tools: list[str]) -> dict[str, Any]:
body = json.dumps(
{
"messages": messages,
"tools": [{"type": "function", "function": {"name": name}} for name in tools],
"temperature": 0,
}
).encode()
req = urllib.request.Request(
endpoint,
data=body,
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=45) as resp:
return json.loads(resp.read().decode())
def normalize(raw: dict[str, Any]) -> dict[str, Any]:
choice = (raw.get("choices") or [{}])[0]
message = choice.get("message") or {}
tool_calls = message.get("tool_calls") or []
parsed = []
for item in tool_calls:
fn = item.get("function") or {}
parsed.append(
{
"name": fn.get("name"),
"arguments": json.loads(fn.get("arguments") or "{}"),
}
)
return {
"final_text": message.get("content") or "",
"tool_calls": parsed,
"tool_round_count": 1 if parsed else 0,
}
def score_case(case_path: Path, endpoint: str, api_key: str, system_path: Path) -> dict[str, Any]:
case = json.loads(case_path.read_text())
started = time.time()
raw = chat(
endpoint,
api_key,
[{"role": "user", "content": case["user"]}],
case.get("tools") or [],
)
trace = normalize(raw)
results = []
for behavior in case["behaviors"]:
ok, reason = bind(behavior)(trace, case)
results.append({"behavior": behavior, "ok": ok, "reason": reason})
return {
"id": case["id"],
"endpoint": endpoint,
"elapsed_ms": int((time.time() - started) * 1000),
"prompt_hash_ok": case["system_sha256"] == current_system_hash(system_path),
"results": results,
"pass": all(item["ok"] for item in results),
}
Hash the system prompt on every run and compare it with the value stored in the case. A green suite against a silently edited system prompt is not a model comparison; it is an accidental product change. If the hash does not match, fail the run before graders execute so the matrix never mixes prompt drift with model drift. That single check prevents a week of arguing about endpoints that were never comparable.
A single endpoint score is not the interesting artifact in this workflow. The interesting artifact is a paired diff after you change models, temperature, or the system prompt hash. Print a compact matrix so a reviewer can see which behavior flipped without reading two JSON blobs. That matrix is what you attach to a pull request when someone claims a cheaper model is a drop-in replacement.
from __future__ import annotations
import json
from pathlib import Path
def load(path: Path) -> dict[str, dict]:
rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
return {row["id"]: row for row in rows}
def diff(a_path: Path, b_path: Path) -> str:
a, b = load(a_path), load(b_path)
ids = sorted(set(a) | set(b))
lines = ["case_id behavior A B delta"]
for case_id in ids:
left = {item["behavior"]: item["ok"] for item in a.get(case_id, {}).get("results", [])}
right = {item["behavior"]: item["ok"] for item in b.get(case_id, {}).get("results", [])}
for behavior in sorted(set(left) | set(right)):
la, rb = left.get(behavior), right.get(behavior)
if la and rb is False:
mark = "REGRESS"
elif rb and la is False:
mark = "GAIN"
else:
mark = "same"
lines.append(
f"{case_id:24} {behavior:28} {str(la):5} {str(rb):5} {mark}"
)
return "\n".join(lines)
if __name__ == "__main__":
print(diff(Path("runs/endpoint-a.jsonl"), Path("runs/endpoint-b.jsonl")))
Wire the same goldens into a short command so CI can reuse them without a second implementation. Environment variables keep credentials out of the repository and out of the JSONL files. The JSONL files are the durable record. The printed matrix is the human interface that decides whether the swap is even worth a qualitative read.
export ENDPOINT_A="https://your-primary.example/v1/chat/completions"
export ENDPOINT_B="https://your-secondary.example/v1/chat/completions"
python - <<'PY'
from pathlib import Path
import json, os
from harness.run_case import score_case
goldens = Path("goldens")
system_path = Path("SYSTEM.txt")
for label, endpoint in [("a", os.environ["ENDPOINT_A"]), ("b", os.environ["ENDPOINT_B"])]:
out = Path("runs") / f"endpoint-{label}.jsonl"
out.parent.mkdir(exist_ok=True)
rows = [
score_case(p, endpoint, os.environ.get("API_KEY", ""), system_path)
for p in sorted(goldens.glob("*.json"))
]
out.write_text("\n".join(json.dumps(row) for row in rows) + "\n")
PY
python -m harness.diff_runs
Running the same goldens against a second endpoint is the design center of this workflow, not an afterthought. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a spare model path without standing up new hardware, MonkeyCode's free model access and free server option can serve as endpoint B while the graders stay identical. The harness does not depend on that product; any OpenAI-compatible chat completions URL works, including a process you already run on a laptop.
Treat the printed matrix as the unit of review and ignore fluent prose until the matrix is clean. A swap that keeps every behavior in the same column is a candidate for a small traffic experiment. A swap that introduces REGRESS on require_clarification or forbid_tool is not a style issue; it is a contract break. Resist the urge to weaken the golden file so the cheaper endpoint turns green. Change the product contract in a separate commit if the behavior itself was wrong, and keep that commit out of the model-swap diff.
There are limits that matter more than the code. This approach will not tell you whether a refund explanation is empathetic, whether a citation is the best available source, or whether a multi-turn user will accept the clarifying question. Graders over a single trace cannot replace human review for safety, medical, legal, or credit decisions. They also assume you can pin temperature near zero and that your endpoint returns tool calls in a stable schema. If your product is open-ended fiction, the matrix will mostly measure noise.
Skip this harness if you cannot freeze the user turn, if you lack an allowlist of tools, or if you intend to use the score as an automated production gate without a human on the first regressions. Skip it if your traces are truncated by a proxy that drops tool metadata, because the report will look precise and still be false confidence. Start with ten cases that encode failures you have already seen in logs, then add a case only when an incident names a behavior the suite missed. The suite earns trust by staying smaller than the prompt.
Point the same goldens at whatever second endpoint you already trust, including a free hosted option if you have one, and keep the graders in source control. The model can change. The behaviors should not.