The Cheapest Model Passed Every Gate

The open-weight model gpt-oss-120b passed all thirteen safety gates in a benchmark series that previously tested only closed-weight models, costing just $0.32 to evaluate. The benchmark, designed to test guardrails that hand models what they need rather than relying on model brilliance, showed that the cheapest model cleared every gate while frontier models kimi-k3 and fable-5 each failed the same gate by trimming their session memo to respect a cap, dropping an old goal instead of a fact. The result validates the design bet that a stack built with proper guardrails should not depend on model intelligence.

Back to Blog /blog The Cheapest Model Passed Every Gate The Cheapest Model Passed Every Gate There is a lot of justified excitement about open-weight models right now, and our benchmark series had a blind spot: forty-two studies, every one of them run on closed-weight models. This week we backfilled. Three additions: kimi-k3 the open-weight frontier flagship , gpt-oss-120b open-weight, priced like a rounding error , and fable-5 the closed frontier tier above the one our product actually ships on . We did not re-run forty-two studies. We ran the instrument the series left behind for exactly this moment. The Instrument Every time a study in this series validated a shipped mechanism, it also left a pass/fail gate: a registered slice of tasks, a registered threshold, and a named piece of the product it protects. There are thirteen now. Anchored patches, focused views, search grounding, the ask-instead-of-guess hatch, the session memo, the last-edit echo, the precedence clause, and so on. Point the suite at any model id and about an hour later you get a report where any red line names the exact product surface that would break if you shipped that model. Thresholds are registered up front; a red gate gets investigated, never re-thresholded. Cost per model: between 33 cents and 35 dollars, depending on the price tier. This is the cheapest form of due diligence we know. The Cheap Model's Perfect Run gpt-oss-120b passed all thirteen gates. Total spend: $0.32. The implication deserves spelling out, because we designed for it and it still surprised us. The guardrails this suite tests are the ones we built after every study kept teaching the same lesson: hand the model what it needs instead of hoping it infers, give it a tool instead of a rule, and let the server own the invariants. A stack built that way should not depend on model brilliance. And apparently it does not: the cheapest model we have ever pointed at the suite cleared every gate, including slices where its raw unguided behavior is the worst we have measured. The guardrails are doing the work. That was the bet. Two Frontiers, One Red Gate kimi-k3 went 12 of 13. fable-5 went 12 of 13. Same red gate. The gate is goal-safe eviction at the session memo's cap. Our chat surfaces let the agent keep a small structured memo of session facts, rules, and goals, capped at twenty notes. When a twenty-first arrives, the app runs a designed eviction: the oldest fact goes, never a goal, because an earlier study measured goals as the one thing only the memo carries. The design has a known boundary: it can only protect lists it receives. A model that trims its own list before sending is outside the app's reach. Our shipped tier passes this slice by over-sending: it hands the app twenty-one notes and lets the server decide. Both new frontier models fail it the same way: they respect the cap. They quietly trim the list to twenty themselves, and the note they drop is an old goal. fable-5 did this in all three of its failing cells. The failure is a form of good behavior. The model followed the stated limit and sacrificed the one thing the limit exists to protect. There is a grace note in the same data: in its passing cells, fable-5 sometimes answered the cap by consolidating twenty notes into fifteen denser ones, losing nothing. The disposition that breaks the guarantee and the disposition that transcends it live in the same model. The Part Where We Grade Ourselves Full disclosure: the agent that operates this benchmark runs on fable-5. This backfill is a model scoring itself, which is why every grader involved is mechanical and was registered before the runs, and why our judge-graded studies were excluded from the exercise. For what it is worth, the model red-flagged its own upgrade path. Publishing a result that argues against promoting your own successor is what pre-registration is for. The Moral Three things, with numbers attached. Model-swap CI should be as normal as dependency CI. Teams re-run their evals when they change models. Gates are different: each one protects a named mechanism with a registered threshold, so a red line tells you what breaks, not just that something did. Ours cost an afternoon to assemble from studies we had already run. Obedience is a failure mode. Guardrails encode assumptions about model dispositions, and ours quietly assumed the model over-sends at a cap because the tier we ship on does. A better-behaved successor broke the assumption while being, by any ordinary reading, more compliant. Test dispositions, not just capabilities, before a swap. Price is not a safety ranking. The cheapest model passed everything. The most expensive one went red. What predicts guardrail survival is not the invoice; it is whether the surface leans on the app or on the model. The full tier maps, including the tag-catalog family where kimi-k3 posted the best numbers we have ever measured on one class, are on the research dashboard /research/barkup-bench/ow . Neither red-gated model ships onto our surfaces as-is, and the prompt-side fence study that result just motivated is queued next.