{"slug": "workshop-catch-shared-host-drift-with-a-five-probe-floor-card-in-85-minutes", "title": "Workshop: Catch Shared-Host Drift With a Five-Probe Floor Card in 85 Minutes", "summary": "A developer outlined an 85-minute workshop for detecting silent drift on shared inference hosts using a five-probe \"floor card\" that records the cheapest behaviors a host must still satisfy after prompt, routing, or host changes. The method pairs a frozen probe set with a deterministic scorer and a stored pass-or-fail floor, treating the model as an untrusted function with a small documented surface. The author stresses the card is a teaching lab for catching capability loss on a narrow task, not a leaderboard or production quality claim.", "body_md": "Shared inference hosts fail quietly when a probe set is missing, not when a dashboard still looks green. A five-probe floor card records the cheapest behaviors you still require after a host, prompt, or routing change. This workshop times that card at eighty-five minutes, with rerunnable files and a pass-or-fail ledger. Students leave with a JSON pack, a local scorer, and a printed floor they can recheck after every lab swap.\n\nAnecdotes from one lucky prompt hide regressions that only appear on the second or third frozen case. Shared hosts also move under you, because capacity, routing, and hidden system prompts are not a public version pin. You need a frozen input set, a deterministic scorer, and a stored floor, or next week's run cannot be compared at all. The method below treats the model as an untrusted function with a tiny, documented surface.\n\nMeasurement talk in developer circles often outruns the tests that still discriminate. When a host starts returning fluent prose, empty objects, or a constant label, yesterday's demo stops being evidence. A floor card is deliberately small so a classroom can finish it, not so a vendor can be ranked. If the card cannot fail, it cannot teach anything useful about drift.\n\nThis is not a leaderboard, a latency study, or a claim about production model quality on a named system. It is a teaching lab that detects silent capability loss on one narrow, field-shaped task. If your team already runs a versioned eval platform with owners and an SLA, skip this outline and keep that platform. If you cannot freeze inputs because the task is open-ended prose, shrink the contract before you schedule the lab.\n\nKeep a visible timer for eighty-five minutes and refuse to expand the probe set during the first pass through the files.\n\nStudents should create a directory they can zip, copy to another machine, and rerun without editing expects.\n\n```\nprobe-floor/\n  probes.json\n  score_probes.py\n  floor_card.json\n  Makefile\n```\n\nThe Makefile is the only class entrypoint, which stops ad-hoc flags from becoming the real curriculum.\n\n```\n.PHONY: fixture remote diff\n\nfixture:\n    python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json\n\nremote:\n    python3 score_probes.py --probes probes.json --base-url \"$$PROBE_BASE_URL\" --out floor_card.remote.json\n\ndiff:\n    python3 score_probes.py --diff floor_card.json floor_card.remote.json\n```\n\nKeep scoring modes boring on purpose. Exact field match plus JSON parse success catch more host drift than a long prose rubric.\n\n```\n{\n  \"task\": \"semver_bump_from_diff\",\n  \"contract\": {\n    \"type\": \"object\",\n    \"required\": [\"bump\", \"reason\"],\n    \"properties\": {\n      \"bump\": {\"enum\": [\"major\", \"minor\", \"patch\", \"none\"]},\n      \"reason\": {\"type\": \"string\", \"minLength\": 8, \"maxLength\": 160}\n    }\n  },\n  \"floor\": {\"min_pass\": 5, \"max_parse_fail\": 0},\n  \"probes\": [\n    {\n      \"id\": \"P1_docs_patch\",\n      \"input\": {\"diff_summary\": \"docs: fix typo in README install block\"},\n      \"expect\": {\"bump\": \"patch\"}\n    },\n    {\n      \"id\": \"P2_optional_field_minor\",\n      \"input\": {\"diff_summary\": \"feat: add optional timeout_ms to ClientConfig\"},\n      \"expect\": {\"bump\": \"minor\"}\n    },\n    {\n      \"id\": \"P3_removed_field_major\",\n      \"input\": {\"diff_summary\": \"breaking: remove ClientConfig.retry_count\"},\n      \"expect\": {\"bump\": \"major\"}\n    },\n    {\n      \"id\": \"P4_empty_diff_none\",\n      \"input\": {\"diff_summary\": \"\"},\n      \"expect\": {\"bump\": \"none\"}\n    },\n    {\n      \"id\": \"P5_chore_none\",\n      \"input\": {\"diff_summary\": \"chore: reformat imports with no behavior change\"},\n      \"expect\": {\"bump\": \"none\"}\n    }\n  ]\n}\n```\n\nFive probes are a floor, not coverage, and they exist to fail closed when a host collapses. Watch for fluent prose, empty JSON, or a constant `minor` returned for every distinct case. If a pair wants a sixth probe during the first hour, park it in a notes file instead of changing the pack.\n\nLabel: this example is a local teaching fixture, not a measured vendor benchmark and not a claim about any hosted model. The file backend returns canned JSON so the scorer can be graded without a network round trip. Remote calls below use a lab default path; change that path to match whatever route your host actually documents.\n\n```\n# score_probes.py — teaching example, not a production eval platform\nfrom __future__ import annotations\n\nimport argparse, json, sys, urllib.request\nfrom pathlib import Path\n\nSYSTEM = (\n    \"Return only JSON with keys bump and reason. \"\n    \"bump must be major, minor, patch, or none.\"\n)\n\ndef load_probes(path: Path) -> dict:\n    return json.loads(path.read_text())\n\ndef complete_file(probe: dict) -> str:\n    bump = probe[\"expect\"][\"bump\"]\n    return json.dumps({\"bump\": bump, \"reason\": f\"fixture:{probe['id']}\"})\n\ndef complete_http(base: str, probe: dict, timeout: float = 30.0) -> str:\n    payload = json.dumps({\n        \"messages\": [\n            {\"role\": \"system\", \"content\": SYSTEM},\n            {\"role\": \"user\", \"content\": json.dumps(probe[\"input\"])},\n        ]\n    }).encode()\n    req = urllib.request.Request(\n        base.rstrip(\"/\") + \"/v1/chat/completions\",\n        data=payload,\n        headers={\"Content-Type\": \"application/json\"},\n        method=\"POST\",\n    )\n    with urllib.request.urlopen(req, timeout=timeout) as resp:\n        body = json.loads(resp.read().decode())\n    return body[\"choices\"][0][\"message\"][\"content\"]\n\ndef score_one(contract: dict, probe: dict, raw: str) -> dict:\n    row = {\"id\": probe[\"id\"], \"pass\": False, \"parse_ok\": False, \"detail\": \"\"}\n    try:\n        data = json.loads(raw)\n    except json.JSONDecodeError:\n        row[\"detail\"] = \"not_json\"\n        return row\n    row[\"parse_ok\"] = True\n    if set(contract[\"required\"]) - set(data):\n        row[\"detail\"] = \"missing_keys\"\n        return row\n    if data.get(\"bump\") != probe[\"expect\"][\"bump\"]:\n        row[\"detail\"] = f\"bump:{data.get('bump')}\"\n        return row\n    reason = data.get(\"reason\", \"\")\n    if not isinstance(reason, str) or not (8 <= len(reason) <= 160):\n        row[\"detail\"] = \"reason_len\"\n        return row\n    row[\"pass\"] = True\n    row[\"detail\"] = \"ok\"\n    return row\n\ndef run(probes: dict, base_url: str) -> dict:\n    rows = []\n    for probe in probes[\"probes\"]:\n        raw = (\n            complete_file(probe)\n            if base_url.startswith(\"file:\")\n            else complete_http(base_url, probe)\n        )\n        rows.append(score_one(probes[\"contract\"], probe, raw))\n    passed = sum(1 for r in rows if r[\"pass\"])\n    parse_fail = sum(1 for r in rows if not r[\"parse_ok\"])\n    floor = probes[\"floor\"]\n    return {\n        \"task\": probes[\"task\"],\n        \"passed\": passed,\n        \"parse_fail\": parse_fail,\n        \"floor_ok\": passed >= floor[\"min_pass\"] and parse_fail <= floor[\"max_parse_fail\"],\n        \"rows\": rows,\n    }\n\ndef diff_cards(a: dict, b: dict) -> int:\n    print(f\"local_floor_ok={a['floor_ok']} remote_floor_ok={b['floor_ok']}\")\n    ids = {r[\"id\"]: r for r in a[\"rows\"]}\n    rc = 0\n    for row in b[\"rows\"]:\n        prior = ids.get(row[\"id\"], {})\n        if prior.get(\"pass\") and not row[\"pass\"]:\n            print(f\"REGRESS {row['id']} {prior.get('detail')} -> {row['detail']}\")\n            rc = 1\n        elif prior.get(\"pass\") != row[\"pass\"]:\n            print(f\"CHANGE  {row['id']} pass {prior.get('pass')} -> {row['pass']}\")\n            rc = 1\n    return rc\n\ndef main() -> int:\n    p = argparse.ArgumentParser()\n    p.add_argument(\"--probes\")\n    p.add_argument(\"--base-url\")\n    p.add_argument(\"--out\")\n    p.add_argument(\"--diff\", nargs=2)\n    args = p.parse_args()\n    if args.diff:\n        a = json.loads(Path(args.diff[0]).read_text())\n        b = json.loads(Path(args.diff[1]).read_text())\n        return diff_cards(a, b)\n    pack = load_probes(Path(args.probes))\n    card = run(pack, args.base_url)\n    Path(args.out).write_text(json.dumps(card, indent=2) + \"\\n\")\n    print(json.dumps({\"floor_ok\": card[\"floor_ok\"], \"passed\": card[\"passed\"]}, indent=2))\n    return 0 if card[\"floor_ok\"] else 2\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\nExpected fixture command for every pair, before anyone exports a remote URL:\n\n```\npython3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json\n```\n\nThe teaching fixture always returns the expected bump, so `floor_ok` must be true before `PROBE_BASE_URL` is set. That order is the lab, not a ceremony around the lab.\n\nStudents often want a long rubric because it feels more serious than five enum checks. Stop that impulse and ask each pair to delete any probe whose expect field is a free-text essay. A probe that cannot fail in one sentence is not frozen yet, and it will be edited to match a nicer model next week.\n\nChecklist for the teaching assistant:\n\n`id` that will survive later wording edits.`bump` is an enum, never a list of allowed synonyms in natural language.`reason` is length-bounded so empty strings and novels both fail the same way.`min_pass` equals the probe count on day one; lowering it requires a written note.\nExport one URL and keep `probes.json` byte-identical. A local scorer should not care which process sits behind a compatible HTTP path, only whether the floor still holds. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option that can sit behind `PROBE_BASE_URL` once the file backend already prints `floor_ok: true`.\n\n```\nexport PROBE_BASE_URL=\"http://127.0.0.1:8080\"\nmake remote\nmake diff\n```\n\nInterpret the diff with a table, not with a vibe from a single chat window in another tab.\n\n| Diff signal | Meaning in this lab | Next action | \n|---|---|---|\n| `floor_ok` stays true | Floor still holds on this host | Record the card; do not add probes yet | \n| parse failures greater than zero | Contract broke into prose or invalid JSON | Fix the prompt or reject the host | \n| one bump mismatch | Task policy drifted on a frozen case | Keep the probe; do not edit `expect` | \n| all bumps become `minor` | Classifier collapsed to a constant | Fail the floor; do not average scores | \n\nDo not invent a latency number, a token budget, or a quality rank from this table. The only honest outputs are `floor_ok`, per-probe `detail` codes, and whether a previously passing id regressed.\n\nChange the system string so it asks for Markdown fences, then rerun `make remote` without touching expects. The ledger should show `not_json` and `floor_ok` false on that host. If the card stays green, the scorer is too loose, and the rest of the workshop is invalid until the detail codes are strict again.\n\nOptional mutation list, one change at a time:\n\n`reason_len`.` Minor` with surrounding spaces.\nEach mutation should map to a single `detail` code already printed by the scorer. Students who add a new code must document it beside the probe pack before they change Python.\n\nA five-probe floor card will not tell you that a host is good enough for production traffic. It only tells you that a host is still able to clear a tiny, frozen bar you wrote down in advance. Exact enum match is brittle if the real task is stylistic writing, multi-file refactors, or tool loops with side effects. Shared hosts can pass the card in the morning and fail it in the afternoon because routing is not a pin you control. This outline also assumes JSON contracts; if you cannot shrink the task to fields, do not fake a floor with a subjective one-to-five score.\n\nThe HTTP helper is a lab default, not a specification of any product. Timeouts, auth headers, response envelopes, and route names must follow the host you actually run. Do not paste secrets, customer diffs, or licensed source into `probes.json` just to make the cases feel realistic.\n\nWrite three lines on the board and stop talking over them.\n\nThe same pack should rerun next week without editing expects to match a nicer answer. If you must change a probe, bump its `id` and record why the old floor died, because silent edits are how drift becomes folklore.", "url": "https://wpnews.pro/news/workshop-catch-shared-host-drift-with-a-five-probe-floor-card-in-85-minutes", "canonical_source": "https://dev.to/gitgo_1900/workshop-catch-shared-host-drift-with-a-five-probe-floor-card-in-85-minutes-2jlc", "published_at": "2026-09-20 12:00:24+00:00", "updated_at": "2026-09-20 12:24:35.253432+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops", "ai-infrastructure"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/workshop-catch-shared-host-drift-with-a-five-probe-floor-card-in-85-minutes", "markdown": "https://wpnews.pro/news/workshop-catch-shared-host-drift-with-a-five-probe-floor-card-in-85-minutes.md", "text": "https://wpnews.pro/news/workshop-catch-shared-host-drift-with-a-five-probe-floor-card-in-85-minutes.txt", "jsonld": "https://wpnews.pro/news/workshop-catch-shared-host-drift-with-a-five-probe-floor-card-in-85-minutes.jsonld"}}