Your Free AI Model Changed Overnight. Here's a Snapshot Test Suite That Notices.

A developer built a snapshot regression suite for LLM outputs after a free-tier model was silently updated, degrading results without any code changes. The tool, drift_check.py, records baseline semantic content and compares new responses using embedding cosine similarity, with hard checks for JSON validity and banned phrases. It runs on a schedule to alert developers when a hosted model has drifted from known-good behavior.

A few weeks ago a small automation I run started producing noticeably worse output. Nothing in my code had changed. No dependency updates, no config edits, no prompt tweaks. The only variable left was the model itself — the free tier I was using had been swapped or updated underneath me, and because I had no baseline recorded anywhere, I couldn't even prove it. I just had a vibe that Tuesday's summaries were worse than Friday's. That experience pushed me to build something I should have had from the start: a snapshot regression suite for LLM outputs . Not an evaluation harness for picking a model I wrote about that before , but a tripwire that runs on a schedule and tells me when a model I've already chosen has drifted. This post is that workflow, with runnable code. We treat pinned npm packages and locked Docker digests as table stakes, but most of us consume LLMs as a floating latest tag. Hosted models get silently upgraded, quantized, re-routed, or retired. Free tiers churn even faster — providers rotate what's available, and a model name that worked last month may now resolve to something different. If your prompts are tuned against one behavior, a silent swap is a breaking change you will never see in a changelog. The fix is the same one we apply everywhere else: record known-good behavior and diff against it. Classic snapshot testing fails for LLMs because output is nondeterministic — you can't string-compare prose. So instead of exact matches, I snapshot the semantic content of responses and compare with embedding cosine similarity, with a hard floor for exact requirements JSON validity, required keys, banned phrases . Here's the core of it. drift check.py : python import json import math import os import sys import time import urllib.request BASELINE PATH = "golden/baseline.json" API URL = os.environ "LLM API URL" your OpenAI-compatible endpoint API KEY = os.environ.get "LLM API KEY", "" some free servers don't need one MODEL = os.environ "LLM MODEL" Each probe has a prompt plus hard constraints that must ALWAYS hold. PROBES = { "id": "json extraction", "prompt": "Extract name and date from: 'Invoice from Acme Corp, dated 2024-03-11.' " "Reply with JSON only.", "must be json": True, "required keys": "name", "date" , }, { "id": "tone summary", "prompt": "Summarize in one neutral sentence: 'The deployment failed twice " "before the rollback succeeded.'", "must be json": False, "banned": "unfortunately", "oops" , tone guardrails }, { "id": "code style", "prompt": "Write a Python function that reverses a string. No explanation, code only.", "must be json": False, "required substrings": "def ", "return" , }, def chat prompt: str - str: body = json.dumps { "model": MODEL, "messages": {"role": "user", "content": prompt} , "temperature": 0.0, reduce noise; not a guarantee } .encode req = urllib.request.Request f"{API URL}/v1/chat/completions", data=body, headers={"Content-Type": "application/json", "Authorization": f"Bearer {API KEY}"}, with urllib.request.urlopen req, timeout=60 as r: return json.load r "choices" 0 "message" "content" def hard check probe: dict, output: str - list str : errors = if probe.get "must be json" : try: parsed = json.loads output.strip .removeprefix " json" .removesuffix " " .strip for k in probe.get "required keys", : if k not in parsed: errors.append f"missing key: {k}" except json.JSONDecodeError: errors.append "output is not valid JSON" for phrase in probe.get "banned", : if phrase.lower in output.lower : errors.append f"banned phrase present: {phrase}" for s in probe.get "required substrings", : if s not in output: errors.append f"required substring missing: {s r}" return errors def main : record = {"model": MODEL, "captured at": time.strftime "%Y-%m-%dT%H:%M:%SZ", time.gmtime , "outputs": {}} failed = False baseline = json.load open BASELINE PATH if os.path.exists BASELINE PATH else None for probe in PROBES: out = chat probe "prompt" record "outputs" probe "id" = out errs = hard check probe, out if baseline: old = baseline "outputs" .get probe "id" , "" sim = token overlap old, out cheap stand-in; see note below if sim < 0.55: errs.append f"semantic drift vs baseline similarity={sim:.2f} " status = "FAIL" if errs else "ok" print f" {status} {probe 'id' }" + f" - {errs}" if errs else "" failed = failed or bool errs if not baseline: os.makedirs "golden", exist ok=True json.dump record, open BASELINE PATH, "w" , indent=2 print "No baseline found — recorded one. Re-run to compare." return sys.exit 1 if failed else 0 def token overlap a: str, b: str - float: """Jaccard similarity over word tokens. Crude but zero-dependency. Swap for embedding cosine similarity when you can afford an embed call.""" ta, tb = set a.lower .split , set b.lower .split return len ta & tb / len ta | tb if ta | tb else 1.0 if name == " main ": main Run it once to capture a golden baseline, then run it on a cron or CI schedule against the same model name. A nonzero exit code means either a hard constraint broke or the output drifted semantically — that's your pager. On the similarity function: I used Jaccard word overlap here so the script has zero dependencies. It's crude and will flag paraphrases. If you're doing this seriously, replace token overlap with cosine similarity over embeddings from any embedding endpoint, and raise the threshold to ~0.9. I've labeled the Jaccard version as a placeholder, not a recommendation. A drift tripwire only works if it runs continuously — daily or on every deploy. On paid APIs, running a probe suite every day is a line item that hobby projects and side tooling quietly die over. This is where I've been using MonkeyCode: it offers free model access and a free server option, which means the scheduled check costs me nothing and runs somewhere other than my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The honest caveat: free availability can change, and the model roster rotates — which is, awkwardly, exactly the drift problem above. My suite monitors the endpoint regardless of provider, so if a free option disappears I point LLM API URL elsewhere and keep my baselines. Treat any free tier as ephemeral infrastructure, never as a dependency you can't replace. Not every failure means "model got worse." I use this decision table: | Signal | Likely cause | Action | |---|---|---| | Hard check fails invalid JSON, missing key | Behavior regression or model swap | Alert, investigate immediately | | All probes drift semantically at once | Model was updated or re-routed | Re-run twice to confirm, then re-tune prompts or re-baseline | | One probe drifts, others stable | Prompt was fragile / borderline | Tighten that prompt's constraints; don't re-baseline yet | Everything drifts after you changed prompts | Self-inflicted | Re-baseline after review — this is working as intended | The last row matters: the suite doubles as prompt-change review. If I edit a prompt and the drift check fires, that's the suite telling me the edit had side effects I didn't intend. Pin your prompts, version your baselines, and treat every hosted model as a mutable dependency. A couple hundred lines of stdlib Python turns "I think the model changed" into a diff you can act on. If you've got a drift story — or a better similarity trick that stays dependency-free — I'd genuinely like to hear it in the comments.