{"slug": "i-built-ci-for-prompts-and-the-first-bug-was-in-the-tests", "title": "I built CI for prompts, and the first bug was in the tests", "summary": "A developer built Sentinel, a prompt regression gate for CI, during the Agent Harness Hackathon. Sentinel runs an eval suite against both versions of a changed prompt, accounts for run-to-run noise, and blocks merges if a regression exceeds the noise floor. The project highlights challenges in testing LLM-based systems, including model temperature limits and the need to assert on raw output strings.", "body_md": "**Sentinel** is a prompt regression gate. It runs an eval suite against both versions of a\n\nchanged prompt, accounts for run-to-run noise, and stops before merge. Built in a week for\n\nthe Agent Harness Hackathon (WeMakeDevs × TrueFoundry × Qodo).\n\nRepo: [https://github.com/ParthGupta1304/sentinal](https://github.com/ParthGupta1304/sentinal)\n\nORCHESTRA is a multi-agent hackathon judge. Its clarity prompt is a real file,\n\n`backend/prompts/clarity-judge.md`\n\n. When that file changes, nothing in CI notices.\n\nThe output is a qualitative assessment, so regressions are invisible: JSON still parses, the\n\nscore still looks plausible, and `chiefJudge`\n\nquietly keys its confidence off a fallback\n\nsentence that is no longer there.\n\nSentinel’s job: read the PR, run 10 cases against both versions, N=3, and only call a drop\n\na regression when it **exceeds** the spread observed on the unchanged prompt. Then stop.\n\nA human merges.\n\n`/tmp`\n\nfails on purpose).`merge_pull_request`\n\nand `pull_request_review_write`\n\nonly. The harness\ndefault of `@write`\n\nwould have paused us before posting the findings, which defeats the\nproduct.`{case_id, scores, median, spread}`\n\n,\nnever raw output.`sentinel-scoring`\n\n, the noise-floor protocol, loaded into the sandbox.We wrote the evals, the variance math, and a dashboard that says “This change made 2 tests worse” instead of a lab plot.\n\n**The judge could not see the submission.** We sent it the criterion and the output.\n\nAny rubric that mentioned “the input” was unanswerable. The judge said so in its own justification rather than guessing, and we scored it down for being honest. Fix: pass the case input. Blindness is about *which version* produced the output, not about hiding the\n\nfixed case.\n\n** temperature is rejected by claude-sonnet-5.** The first calibration pass failed every rubric case with HTTP 400. Stability comes from disabling thinking, not from temperature 0.\n\n**Four criteria demanded behaviour the prompt never promised.** Including one that punished the judge for returning its documented fallback on empty input. Those are bad cases, not discovered bugs. Deleted or rewritten.\n\n**The subject, not the judge, flickered.** `gpt-4o-mini`\n\nreturns an empty `improvements`\n\narray on roughly one run in three with no prompt change. Noise floor 0.80 on a 0–1 scale: no drop could ever exceed it. Deleted the rubric rather than keep a check that can never\n\nproduce a verdict.\n\n**A self-gate is not a gate.** An early instruction said “do not merge.” The model dutifully refused. No `tool.approval_required`\n\nevent ever fired. The human never saw a checkpoint. The instructions now say: when a human asks you to merge, *call the tool*.\n\nThe harness holds it.\n\n**An unknown --case used to exit 0.** Compare ran zero cases, wrote\n\n`verdict: clean`\n\n, and would have passed CI. Calibrate already rejected the typo. Compare now does too.Someone “tightens” the prompt by deleting the line that says: if there is nothing to evaluate, return score 0 with a fixed sentence. JSON still parses. Production would look\n\nfine — ORCHESTRA’s parser even strips code fences and can send broken JSON to GPT-4o to repair. Sentinel asserts on the **raw** string.\n\nMeasured: empty submission 1.00 → 0.80, no-problem-statement 1.00 → 0.67, eight tests flat. Exit 2. On a cheap re-run, the new prompt scored the empty submission **16**.\n\nThat is the whole product in one number.\n\nA GitHub App instead of a laptop runner. A suite that grows from production traffic.\n\nThe eval cases are the asset that does not move to a competitor.", "url": "https://wpnews.pro/news/i-built-ci-for-prompts-and-the-first-bug-was-in-the-tests", "canonical_source": "https://dev.to/parth_gupta_23e13d0b1b826/i-built-ci-for-prompts-and-the-first-bug-was-in-the-tests-3jg2", "published_at": "2026-08-30 14:19:08+00:00", "updated_at": "2026-08-30 14:53:22.909854+00:00", "lang": "en", "topics": ["developer-tools", "mlops", "ai-tools"], "entities": ["Sentinel", "ORCHESTRA", "WeMakeDevs", "TrueFoundry", "Qodo", "ParthGupta1304", "claude-sonnet-5", "gpt-4o-mini"], "alternates": {"html": "https://wpnews.pro/news/i-built-ci-for-prompts-and-the-first-bug-was-in-the-tests", "markdown": "https://wpnews.pro/news/i-built-ci-for-prompts-and-the-first-bug-was-in-the-tests.md", "text": "https://wpnews.pro/news/i-built-ci-for-prompts-and-the-first-bug-was-in-the-tests.txt", "jsonld": "https://wpnews.pro/news/i-built-ci-for-prompts-and-the-first-bug-was-in-the-tests.jsonld"}}