Workshop: Catch Shared-Host Drift With a Five-Probe Floor Card in 85 Minutes

A developer outlined an 85-minute workshop for detecting silent drift on shared inference hosts using a five-probe "floor card" that records the cheapest behaviors a host must still satisfy after prompt, routing, or host changes. The method pairs a frozen probe set with a deterministic scorer and a stored pass-or-fail floor, treating the model as an untrusted function with a small documented surface. The author stresses the card is a teaching lab for catching capability loss on a narrow task, not a leaderboard or production quality claim.

Shared inference hosts fail quietly when a probe set is missing, not when a dashboard still looks green. A five-probe floor card records the cheapest behaviors you still require after a host, prompt, or routing change. This workshop times that card at eighty-five minutes, with rerunnable files and a pass-or-fail ledger. Students leave with a JSON pack, a local scorer, and a printed floor they can recheck after every lab swap. Anecdotes from one lucky prompt hide regressions that only appear on the second or third frozen case. Shared hosts also move under you, because capacity, routing, and hidden system prompts are not a public version pin. You need a frozen input set, a deterministic scorer, and a stored floor, or next week's run cannot be compared at all. The method below treats the model as an untrusted function with a tiny, documented surface. Measurement talk in developer circles often outruns the tests that still discriminate. When a host starts returning fluent prose, empty objects, or a constant label, yesterday's demo stops being evidence. A floor card is deliberately small so a classroom can finish it, not so a vendor can be ranked. If the card cannot fail, it cannot teach anything useful about drift. This is not a leaderboard, a latency study, or a claim about production model quality on a named system. It is a teaching lab that detects silent capability loss on one narrow, field-shaped task. If your team already runs a versioned eval platform with owners and an SLA, skip this outline and keep that platform. If you cannot freeze inputs because the task is open-ended prose, shrink the contract before you schedule the lab. Keep a visible timer for eighty-five minutes and refuse to expand the probe set during the first pass through the files. Students should create a directory they can zip, copy to another machine, and rerun without editing expects. probe-floor/ probes.json score probes.py floor card.json Makefile The Makefile is the only class entrypoint, which stops ad-hoc flags from becoming the real curriculum. .PHONY: fixture remote diff fixture: python3 score probes.py --probes probes.json --base-url file://fixtures --out floor card.json remote: python3 score probes.py --probes probes.json --base-url "$$PROBE BASE URL" --out floor card.remote.json diff: python3 score probes.py --diff floor card.json floor card.remote.json Keep scoring modes boring on purpose. Exact field match plus JSON parse success catch more host drift than a long prose rubric. { "task": "semver bump from diff", "contract": { "type": "object", "required": "bump", "reason" , "properties": { "bump": {"enum": "major", "minor", "patch", "none" }, "reason": {"type": "string", "minLength": 8, "maxLength": 160} } }, "floor": {"min pass": 5, "max parse fail": 0}, "probes": { "id": "P1 docs patch", "input": {"diff summary": "docs: fix typo in README install block"}, "expect": {"bump": "patch"} }, { "id": "P2 optional field minor", "input": {"diff summary": "feat: add optional timeout ms to ClientConfig"}, "expect": {"bump": "minor"} }, { "id": "P3 removed field major", "input": {"diff summary": "breaking: remove ClientConfig.retry count"}, "expect": {"bump": "major"} }, { "id": "P4 empty diff none", "input": {"diff summary": ""}, "expect": {"bump": "none"} }, { "id": "P5 chore none", "input": {"diff summary": "chore: reformat imports with no behavior change"}, "expect": {"bump": "none"} } } Five probes are a floor, not coverage, and they exist to fail closed when a host collapses. Watch for fluent prose, empty JSON, or a constant minor returned for every distinct case. If a pair wants a sixth probe during the first hour, park it in a notes file instead of changing the pack. Label: this example is a local teaching fixture, not a measured vendor benchmark and not a claim about any hosted model. The file backend returns canned JSON so the scorer can be graded without a network round trip. Remote calls below use a lab default path; change that path to match whatever route your host actually documents. score probes.py — teaching example, not a production eval platform from future import annotations import argparse, json, sys, urllib.request from pathlib import Path SYSTEM = "Return only JSON with keys bump and reason. " "bump must be major, minor, patch, or none." def load probes path: Path - dict: return json.loads path.read text def complete file probe: dict - str: bump = probe "expect" "bump" return json.dumps {"bump": bump, "reason": f"fixture:{probe 'id' }"} def complete http base: str, probe: dict, timeout: float = 30.0 - str: payload = json.dumps { "messages": {"role": "system", "content": SYSTEM}, {"role": "user", "content": json.dumps probe "input" }, } .encode req = urllib.request.Request base.rstrip "/" + "/v1/chat/completions", data=payload, headers={"Content-Type": "application/json"}, method="POST", with urllib.request.urlopen req, timeout=timeout as resp: body = json.loads resp.read .decode return body "choices" 0 "message" "content" def score one contract: dict, probe: dict, raw: str - dict: row = {"id": probe "id" , "pass": False, "parse ok": False, "detail": ""} try: data = json.loads raw except json.JSONDecodeError: row "detail" = "not json" return row row "parse ok" = True if set contract "required" - set data : row "detail" = "missing keys" return row if data.get "bump" = probe "expect" "bump" : row "detail" = f"bump:{data.get 'bump' }" return row reason = data.get "reason", "" if not isinstance reason, str or not 8 <= len reason <= 160 : row "detail" = "reason len" return row row "pass" = True row "detail" = "ok" return row def run probes: dict, base url: str - dict: rows = for probe in probes "probes" : raw = complete file probe if base url.startswith "file:" else complete http base url, probe rows.append score one probes "contract" , probe, raw passed = sum 1 for r in rows if r "pass" parse fail = sum 1 for r in rows if not r "parse ok" floor = probes "floor" return { "task": probes "task" , "passed": passed, "parse fail": parse fail, "floor ok": passed = floor "min pass" and parse fail <= floor "max parse fail" , "rows": rows, } def diff cards a: dict, b: dict - int: print f"local floor ok={a 'floor ok' } remote floor ok={b 'floor ok' }" ids = {r "id" : r for r in a "rows" } rc = 0 for row in b "rows" : prior = ids.get row "id" , {} if prior.get "pass" and not row "pass" : print f"REGRESS {row 'id' } {prior.get 'detail' } - {row 'detail' }" rc = 1 elif prior.get "pass" = row "pass" : print f"CHANGE {row 'id' } pass {prior.get 'pass' } - {row 'pass' }" rc = 1 return rc def main - int: p = argparse.ArgumentParser p.add argument "--probes" p.add argument "--base-url" p.add argument "--out" p.add argument "--diff", nargs=2 args = p.parse args if args.diff: a = json.loads Path args.diff 0 .read text b = json.loads Path args.diff 1 .read text return diff cards a, b pack = load probes Path args.probes card = run pack, args.base url Path args.out .write text json.dumps card, indent=2 + "\n" print json.dumps {"floor ok": card "floor ok" , "passed": card "passed" }, indent=2 return 0 if card "floor ok" else 2 if name == " main ": sys.exit main Expected fixture command for every pair, before anyone exports a remote URL: python3 score probes.py --probes probes.json --base-url file://fixtures --out floor card.json The teaching fixture always returns the expected bump, so floor ok must be true before PROBE BASE URL is set. That order is the lab, not a ceremony around the lab. Students often want a long rubric because it feels more serious than five enum checks. Stop that impulse and ask each pair to delete any probe whose expect field is a free-text essay. A probe that cannot fail in one sentence is not frozen yet, and it will be edited to match a nicer model next week. Checklist for the teaching assistant: id that will survive later wording edits. bump is an enum, never a list of allowed synonyms in natural language. reason is length-bounded so empty strings and novels both fail the same way. min pass equals the probe count on day one; lowering it requires a written note. Export one URL and keep probes.json byte-identical. A local scorer should not care which process sits behind a compatible HTTP path, only whether the floor still holds. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option that can sit behind PROBE BASE URL once the file backend already prints floor ok: true . export PROBE BASE URL="http://127.0.0.1:8080" make remote make diff Interpret the diff with a table, not with a vibe from a single chat window in another tab. | Diff signal | Meaning in this lab | Next action | |---|---|---| | floor ok stays true | Floor still holds on this host | Record the card; do not add probes yet | | parse failures greater than zero | Contract broke into prose or invalid JSON | Fix the prompt or reject the host | | one bump mismatch | Task policy drifted on a frozen case | Keep the probe; do not edit expect | | all bumps become minor | Classifier collapsed to a constant | Fail the floor; do not average scores | Do not invent a latency number, a token budget, or a quality rank from this table. The only honest outputs are floor ok , per-probe detail codes, and whether a previously passing id regressed. Change the system string so it asks for Markdown fences, then rerun make remote without touching expects. The ledger should show not json and floor ok false on that host. If the card stays green, the scorer is too loose, and the rest of the workshop is invalid until the detail codes are strict again. Optional mutation list, one change at a time: reason len . Minor with surrounding spaces. Each mutation should map to a single detail code already printed by the scorer. Students who add a new code must document it beside the probe pack before they change Python. A five-probe floor card will not tell you that a host is good enough for production traffic. It only tells you that a host is still able to clear a tiny, frozen bar you wrote down in advance. Exact enum match is brittle if the real task is stylistic writing, multi-file refactors, or tool loops with side effects. Shared hosts can pass the card in the morning and fail it in the afternoon because routing is not a pin you control. This outline also assumes JSON contracts; if you cannot shrink the task to fields, do not fake a floor with a subjective one-to-five score. The HTTP helper is a lab default, not a specification of any product. Timeouts, auth headers, response envelopes, and route names must follow the host you actually run. Do not paste secrets, customer diffs, or licensed source into probes.json just to make the cases feel realistic. Write three lines on the board and stop talking over them. The same pack should rerun next week without editing expects to match a nicer answer. If you must change a probe, bump its id and record why the old floor died, because silent edits are how drift becomes folklore.