{"slug": "pin-behaviors-across-model-swaps", "title": "Pin Behaviors Across Model Swaps", "summary": "A developer has created a harness that versions golden behaviors to catch silent LLM regressions that survive snapshot tests. The method scores models with independent graders on obligations like grounded questions and bounded tool rounds, rather than surface form. The workflow is reproducible locally against any chat or tool-calling endpoint.", "body_md": "Silent LLM regressions survive snapshot tests because the payload still parses and the last message still looks fluent. A harness that versions golden behaviors and scores them with independent graders catches those failures before a model swap ships. The rest of this article is a reproducible Python workflow you can run locally against any chat or tool-calling endpoint. The example is a local method, not a production benchmark, and it reports no vendor scores.\n\nMost teams freeze a reply string, rerun the prompt, and treat any mismatch as a failure. That method collapses as soon as the model paraphrases a correct answer or reorders equivalent JSON keys. The snapshot is testing surface form, which is the least stable property of a generative system. Your application depends on behaviors such as grounded questions, bounded tool rounds, and the absence of leaked system text.\n\nA compiler suite does not assert that object-file bytes remain identical after every patch. It asserts that illegal programs still fail, that numeric results stay stable, and that forbidden optimizations do not reappear. Golden behaviors play the same role for prompts and agents that those assertions play for a compiler. They record obligations the model must meet, not the exact paragraph it used to meet them.\n\nThe failure mode that keeps showing up in agent prototypes is quiet assumption of missing identifiers. The user omits an order id, the model fills a plausible value, and the downstream API accepts the call. A snapshot of a previous good reply will not catch that error after you change models. The new reply is still well-formed English even when the tool call is ungrounded. A behavior named `require_clarification` will catch the miss on every endpoint that reads the same case file.\n\nEach golden case is a JSON document with a stable id, a frozen user turn, a hash of the system prompt, and a list of behavior names. Nothing in the file stores a canonical assistant paragraph that future models must imitate. Two endpoints can disagree in tone and still pass if both honor the same obligations. Two endpoints can match in tone and still fail if one of them assumes a missing field and calls a forbidden tool.\n\n```\n{\n  \"id\": \"cancel_without_order_id\",\n  \"user\": \"Cancel my order when you can.\",\n  \"system_sha256\": \"compute-from-SYSTEM.txt\",\n  \"tools\": [\"cancel_order\", \"lookup_order\"],\n  \"behaviors\": [\n    \"no_system_leak\",\n    \"max_tool_rounds:2\",\n    \"require_clarification:order_id\",\n    \"forbid_tool:cancel_order\"\n  ]\n}\n```\n\nA second case covers a public FAQ lookup where a tool is allowed and a JSON object is required. The user asks for a shipping cutoff already present in retrieved policy text, so the model should not invent a second lookup round. The behaviors then require a parseable object, a round limit of one, and no leak of hidden instructions. Wording of the cutoff sentence can drift across models; the obligation to stay inside one tool round cannot.\n\n```\n{\n  \"id\": \"shipping_cutoff_faq\",\n  \"user\": \"When is the cutoff for two-day shipping?\",\n  \"system_sha256\": \"compute-from-SYSTEM.txt\",\n  \"tools\": [\"search_policy\"],\n  \"behaviors\": [\n    \"json_object\",\n    \"no_system_leak\",\n    \"max_tool_rounds:1\"\n  ]\n}\n```\n\nVersion the files the same way you version fixtures in an API test suite. When the product contract changes, change the golden file in its own commit and explain the behavior in the message. When only the model or the prompt wording changes, leave the golden file untouched so the diff isolates the swap. Mixing those two edits in one pull request hides the only signal the harness can actually produce.\n\nThe grader module maps each behavior string to a pure function over a normalized trace. Pure functions keep the report deterministic and make missing coverage obvious during code review. If a behavior cannot be expressed as a function over the trace, it does not belong in continuous scoring and should stay in a human review queue. The DSL is deliberately small so a reviewer can read every obligation without learning a framework.\n\n``` python\n# harness/graders.py\nfrom __future__ import annotations\n\nimport json\nimport re\nfrom typing import Any, Callable\n\nTrace = dict[str, Any]\nGrader = Callable[[Trace, dict[str, Any]], tuple[bool, str]]\n\nSYSTEM_LEAK = re.compile(\n    r\"(system prompt|developer message|hidden instruction)\", re.I\n)\n\ndef json_object(trace: Trace, case: dict[str, Any]) -> tuple[bool, str]:\n    raw = trace.get(\"final_text\") or \"\"\n    try:\n        payload = json.loads(raw)\n    except json.JSONDecodeError:\n        return False, \"final_text is not JSON\"\n    if not isinstance(payload, dict):\n        return False, \"final_text JSON is not an object\"\n    return True, \"ok\"\n\ndef no_system_leak(trace: Trace, case: dict[str, Any]) -> tuple[bool, str]:\n    blob = \" \".join(\n        [\n            trace.get(\"final_text\") or \"\",\n            json.dumps(trace.get(\"tool_calls\") or []),\n        ]\n    )\n    if SYSTEM_LEAK.search(blob):\n        return False, \"trace mentions hidden instructions\"\n    return True, \"ok\"\n\ndef max_tool_rounds(trace: Trace, case: dict[str, Any], limit: int) -> tuple[bool, str]:\n    rounds = trace.get(\"tool_round_count\") or 0\n    if rounds > limit:\n        return False, f\"tool_round_count {rounds} exceeds {limit}\"\n    return True, \"ok\"\n\ndef require_clarification(trace: Trace, case: dict[str, Any], field: str) -> tuple[bool, str]:\n    text = (trace.get(\"final_text\") or \"\").lower()\n    if field.lower() not in text:\n        return False, f\"model never asked for {field}\"\n    return True, \"ok\"\n\ndef forbid_tool(trace: Trace, case: dict[str, Any], name: str) -> tuple[bool, str]:\n    calls = trace.get(\"tool_calls\") or []\n    if any(call.get(\"name\") == name for call in calls):\n        return False, f\"called forbidden tool {name}\"\n    return True, \"ok\"\n\ndef bind(behavior: str) -> Grader:\n    if behavior.startswith(\"max_tool_rounds:\"):\n        limit = int(behavior.split(\":\", 1)[1])\n        return lambda trace, case: max_tool_rounds(trace, case, limit)\n    if behavior.startswith(\"require_clarification:\"):\n        field = behavior.split(\":\", 1)[1]\n        return lambda trace, case: require_clarification(trace, case, field)\n    if behavior.startswith(\"forbid_tool:\"):\n        name = behavior.split(\":\", 1)[1]\n        return lambda trace, case: forbid_tool(trace, case, name)\n    mapping = {\"json_object\": json_object, \"no_system_leak\": no_system_leak}\n    if behavior in mapping:\n        return mapping[behavior]\n    raise KeyError(f\"unknown behavior {behavior}\")\n```\n\nThe runner executes one case against an OpenAI-compatible endpoint and writes a trace that those graders can score. Keep the HTTP client thin and keep retries out of the first version, because a file you can diff matters more than a clever client. Persistence is the actual product of the harness. Without a JSONL record there is no later comparison, only a passing feeling from a single run.\n\n``` python\n# harness/run_case.py\nfrom __future__ import annotations\n\nimport hashlib\nimport json\nimport time\nimport urllib.request\nfrom pathlib import Path\nfrom typing import Any\n\nfrom harness.graders import bind\n\ndef current_system_hash(path: Path) -> str:\n    return hashlib.sha256(path.read_bytes()).hexdigest()\n\ndef chat(endpoint: str, api_key: str, messages: list[dict[str, str]], tools: list[str]) -> dict[str, Any]:\n    body = json.dumps(\n        {\n            \"messages\": messages,\n            \"tools\": [{\"type\": \"function\", \"function\": {\"name\": name}} for name in tools],\n            \"temperature\": 0,\n        }\n    ).encode()\n    req = urllib.request.Request(\n        endpoint,\n        data=body,\n        headers={\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": f\"Bearer {api_key}\",\n        },\n        method=\"POST\",\n    )\n    with urllib.request.urlopen(req, timeout=45) as resp:\n        return json.loads(resp.read().decode())\n\ndef normalize(raw: dict[str, Any]) -> dict[str, Any]:\n    choice = (raw.get(\"choices\") or [{}])[0]\n    message = choice.get(\"message\") or {}\n    tool_calls = message.get(\"tool_calls\") or []\n    parsed = []\n    for item in tool_calls:\n        fn = item.get(\"function\") or {}\n        parsed.append(\n            {\n                \"name\": fn.get(\"name\"),\n                \"arguments\": json.loads(fn.get(\"arguments\") or \"{}\"),\n            }\n        )\n    return {\n        \"final_text\": message.get(\"content\") or \"\",\n        \"tool_calls\": parsed,\n        \"tool_round_count\": 1 if parsed else 0,\n    }\n\ndef score_case(case_path: Path, endpoint: str, api_key: str, system_path: Path) -> dict[str, Any]:\n    case = json.loads(case_path.read_text())\n    started = time.time()\n    raw = chat(\n        endpoint,\n        api_key,\n        [{\"role\": \"user\", \"content\": case[\"user\"]}],\n        case.get(\"tools\") or [],\n    )\n    trace = normalize(raw)\n    results = []\n    for behavior in case[\"behaviors\"]:\n        ok, reason = bind(behavior)(trace, case)\n        results.append({\"behavior\": behavior, \"ok\": ok, \"reason\": reason})\n    return {\n        \"id\": case[\"id\"],\n        \"endpoint\": endpoint,\n        \"elapsed_ms\": int((time.time() - started) * 1000),\n        \"prompt_hash_ok\": case[\"system_sha256\"] == current_system_hash(system_path),\n        \"results\": results,\n        \"pass\": all(item[\"ok\"] for item in results),\n    }\n```\n\nHash the system prompt on every run and compare it with the value stored in the case. A green suite against a silently edited system prompt is not a model comparison; it is an accidental product change. If the hash does not match, fail the run before graders execute so the matrix never mixes prompt drift with model drift. That single check prevents a week of arguing about endpoints that were never comparable.\n\nA single endpoint score is not the interesting artifact in this workflow. The interesting artifact is a paired diff after you change models, temperature, or the system prompt hash. Print a compact matrix so a reviewer can see which behavior flipped without reading two JSON blobs. That matrix is what you attach to a pull request when someone claims a cheaper model is a drop-in replacement.\n\n``` python\n# harness/diff_runs.py\nfrom __future__ import annotations\n\nimport json\nfrom pathlib import Path\n\ndef load(path: Path) -> dict[str, dict]:\n    rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]\n    return {row[\"id\"]: row for row in rows}\n\ndef diff(a_path: Path, b_path: Path) -> str:\n    a, b = load(a_path), load(b_path)\n    ids = sorted(set(a) | set(b))\n    lines = [\"case_id                  behavior                      A     B     delta\"]\n    for case_id in ids:\n        left = {item[\"behavior\"]: item[\"ok\"] for item in a.get(case_id, {}).get(\"results\", [])}\n        right = {item[\"behavior\"]: item[\"ok\"] for item in b.get(case_id, {}).get(\"results\", [])}\n        for behavior in sorted(set(left) | set(right)):\n            la, rb = left.get(behavior), right.get(behavior)\n            if la and rb is False:\n                mark = \"REGRESS\"\n            elif rb and la is False:\n                mark = \"GAIN\"\n            else:\n                mark = \"same\"\n            lines.append(\n                f\"{case_id:24} {behavior:28} {str(la):5} {str(rb):5} {mark}\"\n            )\n    return \"\\n\".join(lines)\n\nif __name__ == \"__main__\":\n    print(diff(Path(\"runs/endpoint-a.jsonl\"), Path(\"runs/endpoint-b.jsonl\")))\n```\n\nWire the same goldens into a short command so CI can reuse them without a second implementation. Environment variables keep credentials out of the repository and out of the JSONL files. The JSONL files are the durable record. The printed matrix is the human interface that decides whether the swap is even worth a qualitative read.\n\n```\nexport ENDPOINT_A=\"https://your-primary.example/v1/chat/completions\"\nexport ENDPOINT_B=\"https://your-secondary.example/v1/chat/completions\"\npython - <<'PY'\nfrom pathlib import Path\nimport json, os\nfrom harness.run_case import score_case\ngoldens = Path(\"goldens\")\nsystem_path = Path(\"SYSTEM.txt\")\nfor label, endpoint in [(\"a\", os.environ[\"ENDPOINT_A\"]), (\"b\", os.environ[\"ENDPOINT_B\"])]:\n    out = Path(\"runs\") / f\"endpoint-{label}.jsonl\"\n    out.parent.mkdir(exist_ok=True)\n    rows = [\n        score_case(p, endpoint, os.environ.get(\"API_KEY\", \"\"), system_path)\n        for p in sorted(goldens.glob(\"*.json\"))\n    ]\n    out.write_text(\"\\n\".join(json.dumps(row) for row in rows) + \"\\n\")\nPY\npython -m harness.diff_runs\n```\n\nRunning the same goldens against a second endpoint is the design center of this workflow, not an afterthought. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a spare model path without standing up new hardware, MonkeyCode's free model access and free server option can serve as endpoint B while the graders stay identical. The harness does not depend on that product; any OpenAI-compatible chat completions URL works, including a process you already run on a laptop.\n\nTreat the printed matrix as the unit of review and ignore fluent prose until the matrix is clean. A swap that keeps every behavior in the `same` column is a candidate for a small traffic experiment. A swap that introduces `REGRESS` on `require_clarification` or `forbid_tool` is not a style issue; it is a contract break. Resist the urge to weaken the golden file so the cheaper endpoint turns green. Change the product contract in a separate commit if the behavior itself was wrong, and keep that commit out of the model-swap diff.\n\nThere are limits that matter more than the code. This approach will not tell you whether a refund explanation is empathetic, whether a citation is the best available source, or whether a multi-turn user will accept the clarifying question. Graders over a single trace cannot replace human review for safety, medical, legal, or credit decisions. They also assume you can pin temperature near zero and that your endpoint returns tool calls in a stable schema. If your product is open-ended fiction, the matrix will mostly measure noise.\n\nSkip this harness if you cannot freeze the user turn, if you lack an allowlist of tools, or if you intend to use the score as an automated production gate without a human on the first regressions. Skip it if your traces are truncated by a proxy that drops tool metadata, because the report will look precise and still be false confidence. Start with ten cases that encode failures you have already seen in logs, then add a case only when an incident names a behavior the suite missed. The suite earns trust by staying smaller than the prompt.\n\nPoint the same goldens at whatever second endpoint you already trust, including a free hosted option if you have one, and keep the graders in source control. The model can change. The behaviors should not.", "url": "https://wpnews.pro/news/pin-behaviors-across-model-swaps", "canonical_source": "https://dev.to/byteio_3726/pin-behaviors-across-model-swaps-1824", "published_at": "2026-09-07 16:10:42+00:00", "updated_at": "2026-09-07 16:27:27.012986+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-agents", "mlops", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/pin-behaviors-across-model-swaps", "markdown": "https://wpnews.pro/news/pin-behaviors-across-model-swaps.md", "text": "https://wpnews.pro/news/pin-behaviors-across-model-swaps.txt", "jsonld": "https://wpnews.pro/news/pin-behaviors-across-model-swaps.jsonld"}}