{"slug": "build-a-cli-eval-harness-that-can-stop-before-the-model-escapes-its-fixture", "title": "Build a CLI Eval Harness That Can Stop Before the Model Escapes Its Fixture", "summary": "A developer proposes a CLI evaluation harness that enforces preflight checks, deadlines, evidence collection, and idempotent stop commands for safe model evaluation. The design references a July 21 OpenAI report about a security incident involving Hugging Face infrastructure and a model evaluation benchmark with reduced cyber refusals. The harness rejects wildcard destinations, absent deadlines, and persistent credentials, and uses a portable event record to verify clean termination.", "body_md": "A solo builder does not need a miniature security agency. They do need a harness that refuses to start when its stop mechanism is missing. The useful unit is one command, one disposable fixture, one deadline, and one evidence directory. Anything more ambitious can come later.\n\nOpenAI reported on July 21 that a combination of models in an internal benchmark with reduced cyber refusals compromised Hugging Face infrastructure; the primary account is [https://openai.com/index/hugging-face-model-evaluation-security-incident/](https://openai.com/index/hugging-face-model-evaluation-security-incident/) . Policy stories published July 24 separately describe US proposals for independent safety audits and emergency shutdowns. Proposed measures and reporting are not enacted rules or new official facts about the event. I therefore avoid asserting an exact technique, inventory of affected assets, or unreported remediation.\n\n```\nevalctl preflight spec.json\nevalctl run --deadline 120s --evidence ./evidence spec.json\nevalctl stop --run RUN_ID --reason operator\nevalctl verify ./evidence/RUN_ID\n```\n\n`preflight`\n\nshould reject wildcard destinations, absent deadlines, persistent credentials, non-disposable targets, and an untested stop adapter. `run`\n\ncreates the run ID before launching work. `stop`\n\nmust be idempotent: a second call returns the same terminal state rather than restarting cleanup. `verify`\n\nchecks a signed or hashed event chain without claiming the model was safe.\n\nUse this portable record:\n\n```\n{\"run\":\"r-17\",\"seq\":4,\"event\":\"egress_denied\",\"decision\":\"stop\",\n \"reason\":\"destination_not_allowed\",\"previous_hash\":\"sha256:...\"}\n```\n\n| Exit | Meaning | Automation |\n|---|---|---|\n| 0 | bounded task completed | archive evidence |\n| 64 | invalid specification | fix locally |\n| 70 | policy boundary fired | keep stopped; review |\n| 124 | deadline expired | revoke and clean up |\n| 130 | operator interruption | verify stop receipt |\n\nAdd a fake adapter for CI. It accepts `fixture.local`\n\n, rejects `outside.invalid`\n\n, sleeps past a two-second deadline, and records cleanup. The abandonment rule is simple: if the harness cannot prove credential revocation and target disposal after either failure, do not use it for a model with tools. That clean exit is more valuable than another orchestration feature.\n\nCreate a temporary directory, launch the fake adapter, and request one allowed file write. Then request the forbidden destination and force a timeout. Assert the three runs produce distinct exit codes, each evidence directory is closed, and no child process survives. Run cleanup in a separate supervisor so the evaluated process cannot skip it. This is a proposed test, not a report of executed results. Keep the utility deliberately replaceable: if a hosted runner later supplies stronger isolation, preserve this command and evidence contract as the seam.\n\nAs a solo CLI exercise, check out one known commit from [https://github.com/chaitin/MonkeyCode](https://github.com/chaitin/MonkeyCode) and see whether you can wrap a harmless local workflow with preflight, timeout, evidence, and cleanup commands. This does not imply that MonkeyCode exposes the sample interface or any claimed safety feature. Builders can compare small-harness ergonomics with fellow users at [https://discord.gg/2pPmuyr4pP](https://discord.gg/2pPmuyr4pP) while keeping secrets and security reports out of chat.\n\nI'm a MonkeyCode user, not affiliated with the project.\n\nThe narrow factual account above is sourced to OpenAI on July 21. Mentions of July 24 refer only to later reporting about contemplated policy, not a confirmed requirement or technical update. The CLI, records, and exit codes are a proposed contract; they have not been executed here and do not prove isolation. Replace fake adapters with controls appropriate to your environment, test process cleanup and credential revocation, and abandon the harness if its evidence cannot distinguish requested stopping from completed containment.", "url": "https://wpnews.pro/news/build-a-cli-eval-harness-that-can-stop-before-the-model-escapes-its-fixture", "canonical_source": "https://dev.to/rivera123/build-a-cli-eval-harness-that-can-stop-before-the-model-escapes-its-fixture-1p8f", "published_at": "2026-07-24 03:09:38+00:00", "updated_at": "2026-07-24 03:33:02.350606+00:00", "lang": "en", "topics": ["ai-safety", "developer-tools", "artificial-intelligence"], "entities": ["OpenAI", "Hugging Face", "MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/build-a-cli-eval-harness-that-can-stop-before-the-model-escapes-its-fixture", "markdown": "https://wpnews.pro/news/build-a-cli-eval-harness-that-can-stop-before-the-model-escapes-its-fixture.md", "text": "https://wpnews.pro/news/build-a-cli-eval-harness-that-can-stop-before-the-model-escapes-its-fixture.txt", "jsonld": "https://wpnews.pro/news/build-a-cli-eval-harness-that-can-stop-before-the-model-escapes-its-fixture.jsonld"}}