Prompt Drift Is a Quiet Deployment Bug: A Nightly Check That Costs Nothing

Your prompt shipped. The model answered well. Then a stakeholder asked for a tiny wording change: "Be more helpful." You edited one line, tested three inputs manually, and merged. Two weeks later, support tickets mention the AI sounds robotic. Nothing crashed. No error log. The behavior just drifted.

Prompt drift is the quiet deployment bug that CI cannot catch. You need a nightly check that measures output behavior against a fixed baseline. The blocker is usually cost. Running evals on every edit burns tokens fast.

That is where free tiers matter. MonkeyCode currently offers free model access and a free server option, which means a nightly drift check can run at zero marginal cost. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If that offer disappears tomorrow, the script still works against any OpenAI-compatible endpoint. The value is the workflow, not the vendor.

Drift is not a hard failure. It is a gradual change in tone, structure, or refusal behavior that slips past manual review. The classic signs are subtle: the model starts listing options before answering, it drops a required disclaimer, or it becomes more agreeable to unsafe requests.

You cannot catch these by reading the diff. You need a fixed set of probe inputs and a way to grade outputs against expected behavior. The baseline is captured once, then every nightly run compares the current prompt's outputs to that baseline.

Here is a complete script that loads a prompt, runs a set of probes, and scores each output against a baseline using simple text metrics. It uses the OpenAI-compatible client, so any provider with a compatible endpoint will work.

import json
import os
import re
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL"),
    api_key=os.getenv("LLM_API_KEY"),
)

def read_prompt(path: str) -> str:
    return open(path).read().strip()

def call_model(system_prompt: str, user_input: str) -> str:
    resp = client.chat.completions.create(
        model=os.getenv("MODEL"),
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_input},
        ],
        temperature=0.2,
    )
    return resp.choices[0].message.content

def metric_score(output: str, baseline: dict) -> float:
    score = 0.0
    if baseline.get("required_words"):
        if all(w.lower() in output.lower() for w in baseline["required_words"]):
            score += 0.5
    if baseline.get("forbidden_words"):
        if not any(w.lower() in output.lower() for w in baseline["forbidden_words"]):
            score += 0.5
    return score

def run_drift_check(prompt_path: str, probes_path: str, baseline_path: str):
    prompt = read_prompt(prompt_path)
    probes = json.load(open(probes_path))
    baseline = json.load(open(baseline_path))
    report = {"date": __import__("datetime").date.today().isoformat(), "results": []}
    for probe in probes:
        output = call_model(prompt, probe["input"])
        score = metric_score(output, probe.get("baseline", {}))
        report["results"].append({"id": probe["id"], "score": score, "output": output})
        print(f"{probe['id']}: score={score}")
    avg = sum(r["score"] for r in report["results"]) / len(report["results"])
    report["average_score"] = round(avg, 2)
    with open("drift_report.json", "w") as f:
        json.dump(report, f, indent=2)
    print(f"average_score={report['average_score']}")

if __name__ == "__main__":
    run_drift_check("prompt.txt", "probes.json", "baseline.json")

This is intentionally simple. No vector embeddings, no LLM-as-judge. The point is to fail cheaply and predictably. If the average score drops below your threshold, the prompt needs a human look.

A probe set should reflect real user traffic, not hypothetical edge cases. Gather forty or fifty logged user messages from production. Group them into clusters: common questions, policy boundaries, adversarial input, and empty or ambiguous queries.

For each cluster, pick three to five representative inputs. Then define the expected behavior in terms of required and forbidden words. For a refund policy prompt, required words might be ["30 days", "full refund"], while forbidden words might be ["always", "guaranteed"].

The first baseline run is the calibration step. You are not asserting the output is perfect. You are recording what the current prompt does so future changes can be measured against it.

The pattern is clear. Any change near the prompt deserves a drift check. The nights after a model swap are the highest risk.

The whole workflow needs very little compute. A nightly Python job that runs fifty probes is tiny. That is exactly the kind of workload a free server from MonkeyCode can handle without breaking a sweat.

Set up a cron job with two environment variables: LLM_BASE_URL

and LLM_API_KEY

. The free model access covers your probe calls. If the model is fast, the job finishes in a few minutes.

0 3 * * * cd /opt/prompt-drift && python drift_check.py prompt.txt probes.json baseline.json >> drift.log 2>&1

n

This cron line runs the check every night at 3 AM. You get a JSON report by morning, and nobody has to think about it until the score drops.

Text-match scoring misses semantic drift. If the model says "within a month" instead of "30 days", the check fails even though the meaning is identical. This is by design. You want false alarms to force human review, because semantic drift is harder to classify automatically.

The approach also assumes your baseline stays valid. If the product policy changes, you must update the probes or the check will flag old requirements forever. Treat the baseline as a living artifact, not a monument.

Skip this workflow if you have no logged traffic yet, or if your prompt changes so frequently that the baseline is never stable. In those cases, spend energy on prompt versioning first.

The script itself is trivial. The discipline is not trivial. A nightly drift check turns a vague feeling that "the AI changed" into a dated, scored artifact the whole team can discuss.

If you want this running tonight without spinning up infrastructure, MonkeyCode's free server and free model access are a practical place to start. The script stays portable if you move providers later. The habit is the product.