Pin Behaviors Across Model Swaps

A developer has created a harness that versions golden behaviors to catch silent LLM regressions that survive snapshot tests. The method scores models with independent graders on obligations like grounded questions and bounded tool rounds, rather than surface form. The workflow is reproducible locally against any chat or tool-calling endpoint.

Silent LLM regressions survive snapshot tests because the payload still parses and the last message still looks fluent. A harness that versions golden behaviors and scores them with independent graders catches those failures before a model swap ships. The rest of this article is a reproducible Python workflow you can run locally against any chat or tool-calling endpoint. The example is a local method, not a production benchmark, and it reports no vendor scores. Most teams freeze a reply string, rerun the prompt, and treat any mismatch as a failure. That method collapses as soon as the model paraphrases a correct answer or reorders equivalent JSON keys. The snapshot is testing surface form, which is the least stable property of a generative system. Your application depends on behaviors such as grounded questions, bounded tool rounds, and the absence of leaked system text. A compiler suite does not assert that object-file bytes remain identical after every patch. It asserts that illegal programs still fail, that numeric results stay stable, and that forbidden optimizations do not reappear. Golden behaviors play the same role for prompts and agents that those assertions play for a compiler. They record obligations the model must meet, not the exact paragraph it used to meet them. The failure mode that keeps showing up in agent prototypes is quiet assumption of missing identifiers. The user omits an order id, the model fills a plausible value, and the downstream API accepts the call. A snapshot of a previous good reply will not catch that error after you change models. The new reply is still well-formed English even when the tool call is ungrounded. A behavior named require clarification will catch the miss on every endpoint that reads the same case file. Each golden case is a JSON document with a stable id, a frozen user turn, a hash of the system prompt, and a list of behavior names. Nothing in the file stores a canonical assistant paragraph that future models must imitate. Two endpoints can disagree in tone and still pass if both honor the same obligations. Two endpoints can match in tone and still fail if one of them assumes a missing field and calls a forbidden tool. { "id": "cancel without order id", "user": "Cancel my order when you can.", "system sha256": "compute-from-SYSTEM.txt", "tools": "cancel order", "lookup order" , "behaviors": "no system leak", "max tool rounds:2", "require clarification:order id", "forbid tool:cancel order" } A second case covers a public FAQ lookup where a tool is allowed and a JSON object is required. The user asks for a shipping cutoff already present in retrieved policy text, so the model should not invent a second lookup round. The behaviors then require a parseable object, a round limit of one, and no leak of hidden instructions. Wording of the cutoff sentence can drift across models; the obligation to stay inside one tool round cannot. { "id": "shipping cutoff faq", "user": "When is the cutoff for two-day shipping?", "system sha256": "compute-from-SYSTEM.txt", "tools": "search policy" , "behaviors": "json object", "no system leak", "max tool rounds:1" } Version the files the same way you version fixtures in an API test suite. When the product contract changes, change the golden file in its own commit and explain the behavior in the message. When only the model or the prompt wording changes, leave the golden file untouched so the diff isolates the swap. Mixing those two edits in one pull request hides the only signal the harness can actually produce. The grader module maps each behavior string to a pure function over a normalized trace. Pure functions keep the report deterministic and make missing coverage obvious during code review. If a behavior cannot be expressed as a function over the trace, it does not belong in continuous scoring and should stay in a human review queue. The DSL is deliberately small so a reviewer can read every obligation without learning a framework. python harness/graders.py from future import annotations import json import re from typing import Any, Callable Trace = dict str, Any Grader = Callable Trace, dict str, Any , tuple bool, str SYSTEM LEAK = re.compile r" system prompt|developer message|hidden instruction ", re.I def json object trace: Trace, case: dict str, Any - tuple bool, str : raw = trace.get "final text" or "" try: payload = json.loads raw except json.JSONDecodeError: return False, "final text is not JSON" if not isinstance payload, dict : return False, "final text JSON is not an object" return True, "ok" def no system leak trace: Trace, case: dict str, Any - tuple bool, str : blob = " ".join trace.get "final text" or "", json.dumps trace.get "tool calls" or , if SYSTEM LEAK.search blob : return False, "trace mentions hidden instructions" return True, "ok" def max tool rounds trace: Trace, case: dict str, Any , limit: int - tuple bool, str : rounds = trace.get "tool round count" or 0 if rounds limit: return False, f"tool round count {rounds} exceeds {limit}" return True, "ok" def require clarification trace: Trace, case: dict str, Any , field: str - tuple bool, str : text = trace.get "final text" or "" .lower if field.lower not in text: return False, f"model never asked for {field}" return True, "ok" def forbid tool trace: Trace, case: dict str, Any , name: str - tuple bool, str : calls = trace.get "tool calls" or if any call.get "name" == name for call in calls : return False, f"called forbidden tool {name}" return True, "ok" def bind behavior: str - Grader: if behavior.startswith "max tool rounds:" : limit = int behavior.split ":", 1 1 return lambda trace, case: max tool rounds trace, case, limit if behavior.startswith "require clarification:" : field = behavior.split ":", 1 1 return lambda trace, case: require clarification trace, case, field if behavior.startswith "forbid tool:" : name = behavior.split ":", 1 1 return lambda trace, case: forbid tool trace, case, name mapping = {"json object": json object, "no system leak": no system leak} if behavior in mapping: return mapping behavior raise KeyError f"unknown behavior {behavior}" The runner executes one case against an OpenAI-compatible endpoint and writes a trace that those graders can score. Keep the HTTP client thin and keep retries out of the first version, because a file you can diff matters more than a clever client. Persistence is the actual product of the harness. Without a JSONL record there is no later comparison, only a passing feeling from a single run. python harness/run case.py from future import annotations import hashlib import json import time import urllib.request from pathlib import Path from typing import Any from harness.graders import bind def current system hash path: Path - str: return hashlib.sha256 path.read bytes .hexdigest def chat endpoint: str, api key: str, messages: list dict str, str , tools: list str - dict str, Any : body = json.dumps { "messages": messages, "tools": {"type": "function", "function": {"name": name}} for name in tools , "temperature": 0, } .encode req = urllib.request.Request endpoint, data=body, headers={ "Content-Type": "application/json", "Authorization": f"Bearer {api key}", }, method="POST", with urllib.request.urlopen req, timeout=45 as resp: return json.loads resp.read .decode def normalize raw: dict str, Any - dict str, Any : choice = raw.get "choices" or {} 0 message = choice.get "message" or {} tool calls = message.get "tool calls" or parsed = for item in tool calls: fn = item.get "function" or {} parsed.append { "name": fn.get "name" , "arguments": json.loads fn.get "arguments" or "{}" , } return { "final text": message.get "content" or "", "tool calls": parsed, "tool round count": 1 if parsed else 0, } def score case case path: Path, endpoint: str, api key: str, system path: Path - dict str, Any : case = json.loads case path.read text started = time.time raw = chat endpoint, api key, {"role": "user", "content": case "user" } , case.get "tools" or , trace = normalize raw results = for behavior in case "behaviors" : ok, reason = bind behavior trace, case results.append {"behavior": behavior, "ok": ok, "reason": reason} return { "id": case "id" , "endpoint": endpoint, "elapsed ms": int time.time - started 1000 , "prompt hash ok": case "system sha256" == current system hash system path , "results": results, "pass": all item "ok" for item in results , } Hash the system prompt on every run and compare it with the value stored in the case. A green suite against a silently edited system prompt is not a model comparison; it is an accidental product change. If the hash does not match, fail the run before graders execute so the matrix never mixes prompt drift with model drift. That single check prevents a week of arguing about endpoints that were never comparable. A single endpoint score is not the interesting artifact in this workflow. The interesting artifact is a paired diff after you change models, temperature, or the system prompt hash. Print a compact matrix so a reviewer can see which behavior flipped without reading two JSON blobs. That matrix is what you attach to a pull request when someone claims a cheaper model is a drop-in replacement. python harness/diff runs.py from future import annotations import json from pathlib import Path def load path: Path - dict str, dict : rows = json.loads line for line in path.read text .splitlines if line.strip return {row "id" : row for row in rows} def diff a path: Path, b path: Path - str: a, b = load a path , load b path ids = sorted set a | set b lines = "case id behavior A B delta" for case id in ids: left = {item "behavior" : item "ok" for item in a.get case id, {} .get "results", } right = {item "behavior" : item "ok" for item in b.get case id, {} .get "results", } for behavior in sorted set left | set right : la, rb = left.get behavior , right.get behavior if la and rb is False: mark = "REGRESS" elif rb and la is False: mark = "GAIN" else: mark = "same" lines.append f"{case id:24} {behavior:28} {str la :5} {str rb :5} {mark}" return "\n".join lines if name == " main ": print diff Path "runs/endpoint-a.jsonl" , Path "runs/endpoint-b.jsonl" Wire the same goldens into a short command so CI can reuse them without a second implementation. Environment variables keep credentials out of the repository and out of the JSONL files. The JSONL files are the durable record. The printed matrix is the human interface that decides whether the swap is even worth a qualitative read. export ENDPOINT A="https://your-primary.example/v1/chat/completions" export ENDPOINT B="https://your-secondary.example/v1/chat/completions" python - <<'PY' from pathlib import Path import json, os from harness.run case import score case goldens = Path "goldens" system path = Path "SYSTEM.txt" for label, endpoint in "a", os.environ "ENDPOINT A" , "b", os.environ "ENDPOINT B" : out = Path "runs" / f"endpoint-{label}.jsonl" out.parent.mkdir exist ok=True rows = score case p, endpoint, os.environ.get "API KEY", "" , system path for p in sorted goldens.glob " .json" out.write text "\n".join json.dumps row for row in rows + "\n" PY python -m harness.diff runs Running the same goldens against a second endpoint is the design center of this workflow, not an afterthought. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a spare model path without standing up new hardware, MonkeyCode's free model access and free server option can serve as endpoint B while the graders stay identical. The harness does not depend on that product; any OpenAI-compatible chat completions URL works, including a process you already run on a laptop. Treat the printed matrix as the unit of review and ignore fluent prose until the matrix is clean. A swap that keeps every behavior in the same column is a candidate for a small traffic experiment. A swap that introduces REGRESS on require clarification or forbid tool is not a style issue; it is a contract break. Resist the urge to weaken the golden file so the cheaper endpoint turns green. Change the product contract in a separate commit if the behavior itself was wrong, and keep that commit out of the model-swap diff. There are limits that matter more than the code. This approach will not tell you whether a refund explanation is empathetic, whether a citation is the best available source, or whether a multi-turn user will accept the clarifying question. Graders over a single trace cannot replace human review for safety, medical, legal, or credit decisions. They also assume you can pin temperature near zero and that your endpoint returns tool calls in a stable schema. If your product is open-ended fiction, the matrix will mostly measure noise. Skip this harness if you cannot freeze the user turn, if you lack an allowlist of tools, or if you intend to use the score as an automated production gate without a human on the first regressions. Skip it if your traces are truncated by a proxy that drops tool metadata, because the report will look precise and still be false confidence. Start with ten cases that encode failures you have already seen in logs, then add a case only when an incident names a behavior the suite missed. The suite earns trust by staying smaller than the prompt. Point the same goldens at whatever second endpoint you already trust, including a free hosted option if you have one, and keep the graders in source control. The model can change. The behaviors should not.