The gate said pass. The model never finished its sentence. This postmortem covers a patch gate that streamed a reviewer verdict from a model, then parsed it with a repair step. A cut stream produced a valid, complete, wrong decision. The failure is reproducible in under two minutes with the code bel
ow. Scope note: the pipeline here is a reconstructed lab reproduction, not a vendor incident report. Every claim below is checked by the included test. Impact: one AI-authored patch reached the protected branch without a full review pass.
Duration: the gate ran with the faulty parser for 19 days. Detection: a re-run on a different day returned fail for the same diff. The verdict was not stable. Blast radius: the bad patch itself was small.
The dangerous part was the certificate attached to it. T+0 — A patch job submits a diff to the reviewer call. The gate streams the response. T+0.8s — An intermediary closes the connection early.
The stream ends mid-object. T+0.9s — json.loads raises. The repair branch appends } and retries. T+0.95s — The repaired string parses.
verdict is "pass". T+1.0s — The gate logs verdict=pass source=repaired and merges. T+19d — A replay on a healthy connection returns "fail" with two blocking findings. T+19d — Parser replaced.
All prior source=repaired records are re-audited by hand. The original parser was four lines long. It looked defensive. import json def old_verdict(raw: str) -> str: try: return json.loads(raw)["verdict"] except json.JSONDecodeError: repaired = raw.rstrip().rstrip(",") + "}" return json.loads(repaired)["verdict"] The repair assumed truncation happens at the end of a well-formed object.
That assumption is false for streams. A cut can land after any field, and field order is not guaranteed. A cut after the first field yields this input: {"verdict": "pass", "diff_sha": "9f2c The repair appends } and produces a parseable object with the verdict intact. Nothing else is required to pass the gate.
The decision policy read one key, so one key was enough. Permissive repair. The catch branch converted an error into a decision instead of a retry. Field order dependence.
verdict was emitted first, so the cheapest field to truncate was the one that mattered. No completeness check. The gate had no byte length, no digest, and no end marker to test against. Single-key policy.
Any parsed object with a verdict key was trusted, regardless of diff_sha or policy version. Free-tier routing. The reviewer call ran on free model access. The gate ran on the free server option, which restarts between jobs.
Neither is inherently unsafe, but both add intermediaries and interrupts the gate did not model. Green dashboards. Every run exited 0. The repair path was counted as success, not as a degraded run.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate now refuses anything it cannot prove is complete. The response must carry a declared byte length, a SHA-256 digest, and a terminal sentinel. The parser fails closed on all three checks.
import hashlib import json import re ENVELOPE = re.compile(rb"^MC1 (?P<len>\d+) (?P<sha>[0-9a-f]{64})\n(?P<body>.*)$", re.S) REQUIRED = ("verdict", "diff_sha", "policy_version") class IncompleteResponse(Exception): pass def parse_envelope(raw: bytes) -> dict: m = ENVELOPE.match(raw) if not m: raise IncompleteResponse("missing envelope header") body = m.group("body") declared = int(m.group("len")) digest = hashlib.sha256(body).hexdigest() if len(body) != declared: raise IncompleteResponse(f"length {len(body)} != {declared}") if digest != m.group("sha").decode(): raise IncompleteResponse("digest mismatch") if not body.endswith(b"\nEND\n"): raise IncompleteResponse("missing sentinel") payload = json.loads(body[:-5]) for key in REQUIRED: if key not in payload: raise IncompleteResponse(f"missing field {key}") return payload Three properties follow. Every one is testable. Truncation changes the digest. A cut stream cannot satisfy the header. Field order stops mattering. The digest covers the whole body. Repair is gone. There is no code path from a parse error to a verdict.
Exhaustive truncation is cheap. Loop over every byte offset of a real envelope and require a rejection at each one. def test_every_truncation_fails_closed(): raw = build_envelope({ "verdict": "pass", "diff_sha": "9f2c", "policy_version": "gate-2026-07", }) assert parse_envelope(raw)["verdict"] == "pass" for cut in range(len(raw)): try: parse_envelope(raw[:cut]) except IncompleteResponse: continue raise AssertionError(f"truncation at byte {cut} was accepted") Add two more cases to the same suite: Byte flip. Mutate one byte in body.
Expect digest mismatch. Sentinel removal. Strip \nEND\n and fix the length and digest. Expect rejection.
A local proxy that cuts the stream at a chosen offset reproduces the incident deterministically. No vendor timing is involved. import socket def serve_truncated(port: int, payload: bytes, cut: int) -> None: srv = socket.socket() srv.bind(("127.0.0.1", port)) srv.listen(1) conn, _ = srv.accept() with conn: conn.recv(65536) conn.sendall(payload[:cut]) srv.close() Run the gate twice. Once with a full payload, once with cut=34.
The old parser returns pass both times. The new parser returns pass only for the full payload. Gate property Free tier is fine Use a paid or self-hosted runner Blocking merge decisions No Yes Advisory lint or summary Yes Either Needs warm caches across runs No Yes Needs sub-second p99 latency No Yes Needs vendor attestation for audit No Yes Reproducing parser failures Yes Either The envelope fix covers truncation and tampering in transit. It does not prove the model reasoned correctly.
A confident pass inside a valid envelope is still a pass. Cold starts and per-job restarts on shared capacity remain. Budget the retry, and make retries visible as their own status line. Free allowances change.
Read the current limits on the product page before sizing a pipeline, and never hardcode them into a policy. Teams that need a contractual SLA on gate latency. Regulated environments that require vendor-side audit records. Workloads that exceed the free allowance at peak, where a hard stop blocks releases.
Anyone who cannot spare the engineering time for exhaustive truncation tests. The parser is a single function with no fallback branch. Every degraded run writes status=incomplete and blocks the merge. A nightly job replays yesterday's diffs on fresh connections and diffs the verdicts.
Verdict instability is now an alert, not a curiosity. If you run AI patch gates on free model access and a free server, try the truncation loop first. It is thirty lines, and it tells you whether your gate fails closed.
Key Takeaways #
- •Scope note: the pipeline here is a reconstructed lab reproduction, not a vendor incident report.
- •This story was reported by Dev.to , covering developments in thedev space.
- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.
📖 Continue reading the full article: