Find Fragile Cases With a Cross-Model Pass Matrix

A developer has published a lightweight, standard-library-only harness that runs the same golden eval cases against multiple models and records a per-case pass vector rather than a single aggregate score, making cross-model disagreements visible before totals. The approach stores each cell's trial results, last output, case fingerprint and grader fingerprint so that a failure can be diagnosed later, and isolates all provider-specific code in a single adapter function so adding a model is a config entry plus one SDK call.

A prompt that passes your eval on one model has told you almost nothing about how it behaves on the next one. The cheapest robustness signal you can collect is a pass matrix: the same golden cases run against several models, one pass vector per case, disagreement read before totals. The vector, not the percentage, is what shows you which cases your prompt only half handles. Model churn is the constant in this work. Every few weeks another endpoint shows up with a different tokenizer, a different refusal posture, and a different opinion about what "concise" means. Teams pin one model, tune a prompt against it for a week, then switch providers when price or latency moves. That switch is a config change, it ships quietly, and nothing in the eval output looks unusual, because the eval only ever knew one model. The runner below keeps a single grader and pushes every provider difference into an adapter function. Cases live in a JSONL file so they can be diffed in review, and each case carries only checks a deterministic grader can evaluate. Inputs are truncated here for length; in your repository they should be the real payloads. {"case id":"refund-window","input":"How long do I have to return an opened item?","must include": "14 days" ,"must not include": "lifetime" } {"case id":"invoice-json","input":"Summarize invoice 88 as JSON with total and currency.","json fields": "total","currency" } {"case id":"empty-cart","input":"My cart is empty, can I check out?","must include": "cart is empty" } python sweep.py - standard library only import hashlib, inspect, json, re, time from datetime import datetime, timezone from itertools import product def norm text : return re.sub r"\s+", " ", text or "" .strip .lower def grade case, output : checks = for needle in case.get "must include", : checks.append "include:" + needle, norm needle in norm output for needle in case.get "must not include", : checks.append "exclude:" + needle, norm needle not in norm output if "json fields" in case: try: payload = json.loads output ok = all field in payload for field in case "json fields" except json.JSONDecodeError: ok = False checks.append "json fields", ok return checks GRADER FP = hashlib.sha256 inspect.getsource grade .encode .hexdigest :12 def fingerprint obj : return hashlib.sha256 json.dumps obj, sort keys=True .encode .hexdigest :12 The adapter is the only provider-specific code in the whole harness, so adding a model means adding a config entry and one SDK call. The stub below is a contract, not a working client; it is deliberately unexecuted here, because credentials and SDK names are yours to choose. python def complete model id, prompt, params=None : """Implement per provider. Not executed in this article.""" raise NotImplementedError Each cell in the matrix stores its trial results, the last output, the case fingerprint, and the grader fingerprint. That combination is what makes a disagreement diagnosable a week later, when you no longer remember whether the prompt or the grader moved. python def sweep models, cases, trials=3, pause=0.0 : ledger = {} for model, case in product models, cases : results = for in range trials : output = complete model "id" , case "input" , model.get "params", {} results.append all ok for , ok in grade case, output if pause: time.sleep pause ledger model "id" , case "case id" = { "passed": all results , "trials": results, "output": output, "prompt fp": fingerprint case , "grader fp": GRADER FP, "model params": model.get "params", {} , } return ledger def matrix ledger, models, cases, path="matrix.csv" : lines = "case id," + ",".join m "id" for m in models + ",class" for case in cases: vector = "".join "1" if ledger m "id" , case "case id" "passed" else "0" for m in models label = "stable" if set vector == {"1"} else "blocked" if set vector == {"0"} else "fragile" lines.append f"{case 'case id' },{','.join vector },{label}" open path, "w" .write "\n".join lines + "\n" return "\n".join lines Requiring all trials to pass before a cell counts as a pass is a deliberate, blunt choice. It inflates the false-failure rate slightly and shrinks the false-pass rate a lot, which is the direction you want when the matrix is deciding where to spend human attention. A column of ones across every model means the case is stable enough to serve as a smoke test, and it earns a place in your fast suite. A column of zeros across every model is almost never a model problem; check the case text, then the grader, then the prompt, in that order. A mixed column is the interesting one, because it names a specific input where your instruction is ambiguous rather than hard. | Vector | Reading | First action | |---|---|---| | 1111 | stable | keep as smoke case | | 1101 | fragile | inspect outputs, usually grader or ambiguity | | 0000 | blocked | fix case, grader, then prompt | Triage order matters more than tooling here. Suppose refund-window produces 1101 and you discover the failing model answers "fourteen days" while the other three answer "14 days". The grader is doing substring matching on a normalized string, so the digit and the word are different tokens to it, and the disagreement was manufactured by your check rather than discovered in the model. The fix is a canonical form or a small regex alternation, applied once, and then a full re-run with a bumped grader fingerprint so the history stays honest. If the outputs differ substantively instead, with one model adding a cheerful preamble and skipping the number, the ambiguity is in the prompt. Rewrite the instruction to require the number in the first sentence, re-run only the fragile column at first, and promote the case back into the full sweep once its vector is all ones. Blocked cases deserve the same treatment before they quietly rot in the suite. The arithmetic is simple and worth doing before you start: models times cases times trials. Twenty-four cases, four models, and three trials is 288 calls, which is nothing on a metered API and still worth logging. A wider sweep is where free model access stops being a curiosity, because you can afford to keep several candidate models in the matrix instead of testing the one you already pay for. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Where that matters for this workflow is the boring operational half. MonkeyCode's free model access, as described by the operator, lets you keep a handful of adapters live in the matrix without a billing account, and the free server option means the sweep runner does not need its own provisioned machine. Confirm the current terms, model list, and any limits on the project's own pages before you build a routine around them, since availability terms change and this article cannot track them for you. Substring and JSON-key grading measures surface compliance, not correctness, so a green matrix is evidence about format and constraint adherence and nothing more. Three or four models are a sampled population rather than a ranking, and publishing cross-model leaderboards from twenty cases is statistically empty even when the table looks tidy. Free endpoints can throttle or change without notice, so cache outputs to disk keyed by model, case, and trial, and add backoff rather than retrying tightly. Skip this if your product pins exactly one model with no fallback path, because the matrix will only confirm what you already assume. Skip it too if your grader is still unstable across repeated runs on one model, since you would then be measuring grader noise with extra steps. Fix the single-model baseline first, then widen. Start with four models you would actually ship and fifteen cases you would actually argue about. The first fragile vector usually pays for the whole exercise. If you want to run the sweep without a billing setup, put the runner on MonkeyCode's free server option and check the current model list before you wire up the adapters.