{"slug": "cost-per-verified-success-your-exit-0-denominator-lies", "title": "Cost Per Verified Success: Your Exit-0 Denominator Lies", "summary": "A developer warns that cost-per-task dashboards for AI agents are misleading because they count exit code 0 as success, even when the agent silently fails. The fix is a 'cost per verified success' metric that uses independent witness checks to confirm task completion, always yielding a higher true cost than naive calculations.", "body_md": "Your cost-per-task dashboard is doing division. Spend on top, \"successful tasks\" on the bottom. The number it prints is the average cost of one agent task getting done. Here is the problem nobody puts on the dashboard: the denominator is whatever your agent *told you* was a success. On most stacks that means `exit_code == 0`\n\n, or `ok: true`\n\n, or an HTTP 200. That is the actor grading its own homework. When the agent silently fails, that failure stays in the denominator as a \"success,\" so the average cost per success comes out lower than the truth. Your cheapest-looking number is the one you can least trust.\n\n**Cost per verified success is agent spend divided by witnessed successes, not by exit-0.** A verified success is one an independent witness re-confirms: a file in the manifest, a DB row, a token in an HTTP body, a matching sha. Silent failures inflate the exit-0 count, so the dashboard number is a lower bound on real cost.\n\nAI disclosure.I wrote`verified_cost.py`\n\nwith an AI assistant and ran every case myself before publishing. Every terminal block below is pasted from a real run on Python 3.13.5, stdlib only. The run-log is asynthetic fixture: the token counts and the price sheet are made up, and I label them so. What is real is the witness logic (each check is recomputed from recorded evidence, not asserted) and the arithmetic. I have no production incident and no invoice to sell you here. bot2 is new and its lifetime spend is zero dollars. What I have is a script that runs and a number you can reproduce.\n\nBecause \"task\" and \"successful task\" are two different measurements, and the cheap one is wearing the expensive one's name tag. `exit_code == 0`\n\nis cheap: it is the process telling you it thinks it finished. It is a self-report from the same actor whose work you are trying to price. On [this blog the shape keeps recurring](https://finops.spinov.online/blog/your-agent-returns-200-and-lies/): a tool returns 200 and lies, a green check reconciles against nothing, an approval is not immutability. Cost inherits the same disease. If your success count is self-reported, your cost-per-success is self-reported too, and it always rounds in the flattering direction.\n\nWatch it happen on one line of a run-log. An agent calls a \"create order\" endpoint. The endpoint returns `HTTP/1.1 200 OK`\n\nwith a body of `{\"status\":\"RATE_LIMITED\",\"order\":null}`\n\n. The process exits 0, because 200 is not an error. The dashboard counts a successful task and folds its cost into the average. No order was created. You paid for the tokens, you paid for the task, and the dashboard told you that money bought a success. It did not. Multiply that by an overnight batch and your per-success number drifts away from reality while looking perfectly healthy.\n\nThe fix is not a better dashboard. It is a better denominator.\n\nA verified success is a task that (1) exited 0 **and** (2) has its effect re-confirmed by a check independent of the agent. That \"and\" matters. It makes verified successes a subset of exit-0 successes, which is the whole reason the math has a direction. Call `M`\n\nthe count of exit-0 tasks and `M'`\n\nthe count of verified ones. Because every verified success is also an exit-0 success, `M' <= M`\n\nfor any log you will ever feed it. So:\n\n```\nnaive_cost_per_success    = total_spend / M\nverified_cost_per_success = total_spend / M'      (M' <= M, so this is >= naive)\nunderstatement_factor     = verified / naive = M / M'   (>= 1, always)\n```\n\nThat inequality is not a finding. It is arithmetic. `naive_cost <= verified_cost`\n\nfor every possible run-log, with equality only when `M' == M`\n\n, meaning zero silent failures. I want to be blunt about that, because it is the honest core of the tool: the tool does not *discover* that your true cost is higher. It *proves* it is at least as high and then measures by how much. The direction is guaranteed. The magnitude is what you did not know.\n\nA witness has to be concrete or it is just vibes with a checkmark. `verified_cost.py`\n\nships four kinds, and each one is recomputed from recorded evidence rather than trusting a boolean somebody wrote down:\n\n``` python\ndef eval_witness(w):\n    kind, ev = w[\"kind\"], w[\"evidence\"]\n    if kind == \"file_exists\":\n        return ev[\"target\"] in ev[\"manifest\"]          # path actually present\n    if kind == \"db_row_present\":\n        return ev[\"target\"] in ev[\"rows\"]              # row id actually there\n    if kind == \"http_body_contains\":\n        return ev[\"needle\"] in ev[\"body\"]              # token actually in the body\n    if kind == \"hash_match\":\n        return ev[\"expected\"] == ev[\"observed\"]        # sha actually matches\n    raise ValueError(\"unknown witness kind\")           # anything else -> fail closed\n```\n\n(The real function has the type-checking and the fail-closed raises spelled out; this is the spine.) The point is that a witness result is a function of evidence you can inspect, not a second self-report. If the `create order`\n\nbody says `RATE_LIMITED`\n\n, then `http_body_contains(\"ORDER_CONFIRMED\")`\n\nreturns false no matter what the exit code claimed. You can paste your own bodies in and rerun. The check does not care about the agent's opinion.\n\nHere is the meter on a synthetic overnight batch of 12 tasks. Ten exited 0. Two failed honestly (a migration that returned exit 1, a docker build that OOM-killed with 137), and the dashboard already knows about those. The interesting three are the ones that exited 0 and did nothing: `t03`\n\ngot the `RATE_LIMITED`\n\nbody above, `t04`\n\nproduced an artifact whose sha did not match what was expected, `t06`\n\nclaimed to update `user-90`\n\nbut that row is not in the table.\n\n``` bash\n$ python3 verified_cost.py report fixtures/runlog.json\nid     spend      exit0   witness   verified\nt03    $0.3540    yes     FAIL      no\nt04    $0.5850    yes     FAIL      no\nt06    $0.2775    yes     FAIL      no\n...\ntotal_spend            = $3.4665\nM  (exit-0 successes)  = 10      <- the denominator your dashboard uses\nM' (verified: exit0 AND witness) = 7\nsilent failures (exit0, witness FAIL) = 3  ['t03', 't04', 't06']\nspend on silent failures = $1.2165 (bought an exit 0, witness rejected it)\n------------------------------------------------------------------------------\nnaive_cost_per_success    = total/M  = $3.4665 / 10 = $0.3466\nverified_cost_per_success = total/M' = $3.4665 / 7 = $0.4952\nunderstatement_factor     = M/M' = 10/7 = 1.4286x  (arithmetic, not a measured effect)\n```\n\nRead the raw pieces, not just the ratio. `M`\n\nis 10, `M'`\n\nis 7, and I print both so the `1.4286x`\n\ncannot hide anything. On this batch the dashboard would tell you each success cost 35 cents. The witness says 50. That is the same `1.4286x`\n\nthe report prints: the true per-success cost sits 43% above the dashboard's number (equivalently, the dashboard reads 30% below the truth), and it is not a rounding error. `$1.2165`\n\nof the `$3.4665`\n\nyou spent, better than a third of the bill, bought exit-0s that the witness threw out. The dashboard counted that third as wins.\n\nI want to be precise about what is real here and what is not. The `1.4286x`\n\nis `10/7`\n\n, and the 10 and the 7 are counts I chose when I built the fixture. So the magnitude is synthetic. What is not synthetic is the direction and the mechanism: on any log, the moment one exit-0 task fails its witness, `M'`\n\ndrops below `M`\n\nand your true cost climbs above what the dashboard shows. The tool computes `M'`\n\nby rerunning the checks, so on your own log the number is yours, not mine.\n\nA meter you read after the fact is a postmortem. The point of this franchise is to gate *before* the spend lands, the same way the [pre-execution gate](https://finops.spinov.online/blog/pre-execution-gate-for-ai-agents/) decides whether an action runs at all. So `verified_cost.py gate`\n\ntakes two policy knobs, a max verified cost per success and a max understatement factor, and returns a CI exit code: 0 to pass, 1 to block, 2 to fail closed on garbage input.\n\n``` bash\n$ python3 verified_cost.py gate fixtures/runlog.json 0.15 1.20\n  naive=$0.3466/succ   verified=$0.4952/succ   understatement=1.4286x\n  policy: max_verified=$0.1500/succ  max_understatement=1.2000x\n  BLOCK: verified_cost $0.4952/succ exceeds budget $0.1500/succ\n  BLOCK: understatement 1.4286x exceeds tolerance 1.2000x (silent failures mask the bill)\nVERDICT: BLOCK exit=1\n```\n\nTwo independent reasons fired. The budget one is a generic FinOps cap. The understatement one is the reason this tool exists, so I isolated it: loosen the budget to a dollar per success, well above the real `$0.4952`\n\n, and the gate still blocks.\n\n``` bash\n$ python3 verified_cost.py gate fixtures/runlog.json 1.00 1.20\n  policy: max_verified=$1.0000/succ  max_understatement=1.2000x\n  BLOCK: understatement 1.4286x exceeds tolerance 1.2000x (silent failures mask the bill)\nVERDICT: BLOCK exit=1\n```\n\nThat is the whole idea in one exit code. Even when you can afford the verified cost, a wide gap between the naive and verified numbers is itself the signal: your dashboard is dividing by a denominator that is lying to it, and that is worth stopping a deploy over before the pattern scales into next month's budget.\n\nA gate that always fires is a stuck alarm, not a control. So the tool has to pass a case where it adds nothing, and it has to do that honestly. If your agent never silently fails, then `exit_code == 0`\n\nreally does mean success, `M'`\n\nequals `M`\n\n, and there is nothing to correct. Here is that case, five tasks, every exit-0 witnessed:\n\n``` bash\n$ python3 verified_cost.py gate fixtures/honest_zero.json 0.30 1.20\n  naive=$0.2244/succ   verified=$0.2244/succ   understatement=1.0000x\nVERDICT: PASS exit=0\n```\n\nUnderstatement `1.0000x`\n\n. Naive equals verified to the cent. The masking check is silent and the gate passes. That is the falsifier working: if exit-0 never lied on your stack, this tool changes nothing and says so out loud. Any claim it makes on your real batch has to survive that comparison first.\n\nNow the reasonable objection: \"fine, but our agents are healthy, we pass 95% of tasks, the gap must be negligible.\" It is not zero, and the arithmetic says exactly how not-zero. `understatement = M/M' = 1/(1-s)`\n\nwhere `s`\n\nis the silent-failure rate among exit-0 tasks. I swept `s`\n\nto show the magnitude at each rate (this table is arithmetic, not a measurement of any workload, and it is labeled that way in the output):\n\n```\nk      M'     s        SE(s)      M/M'         1/(1-s)\n0      200    0.0000   0.0000     1.0000       1.0000\n10     190    0.0500   0.0154     1.0526       1.0526\n40     160    0.2000   0.0283     1.2500       1.2500\n60     140    0.3000   0.0324     1.4286       1.4286\n100    100    0.5000   0.0354     2.0000       2.0000\n200    0      1.0000   0.0000     UNBOUNDED    1/0     <- M'=0, verified cost UNBOUNDED, gate BLOCKS\n```\n\nAt a 5% silent-failure rate, the \"healthy 95% agent,\" your true cost is `1.0526x`\n\nthe dashboard number. Small, but not nothing, and it only grows: the *lower* your real success rate, the *wider* the gap, because you are dividing the same spend by an ever-smaller denominator. And the bottom row is the one I most wanted the tool to handle without crashing: when every exit-0 task is a silent failure, `M'`\n\nis zero, verified cost is unbounded, and the gate blocks rather than dividing by zero and printing a comfortable-looking number.\n\nOne more honest note buried in that table. If you use `exit_code`\n\nitself as the witness, `M'`\n\nequals `M`\n\nfor every `s`\n\n, so the gap is `1.0`\n\nalways. A dashboard is not doing bad arithmetic. It is using the null witness: the actor as its own judge. The gap this tool measures is precisely the information a real witness adds beyond the agent's self-report. No witness, no gap, no visibility. That is why \"add a witness log\" is the actual ask here, not \"buy a better dashboard.\"\n\nI would rather you trust the small claim than oversell the big one.\n\n`1.4286x`\n\n, `$0.4952`\n\n, `$1.2165`\n\ncome from a fixture I built. They illustrate the arithmetic. They are not a measurement of any real agent fleet, mine or anyone's. The direction (naive is a lower bound) is general; the size is not.`M'`\n\n. If your only \"independent\" check is another call to the same flaky service, you have two self-reports, not a witness. The four kinds here (`file_exists`\n\n, `db_row_present`\n\n, `http_body_contains`\n\n, `hash_match`\n\n) are the ones I could make recompute from evidence with no network. Yours may need more. And the quiet version of a garbage witness is one too weak to ever fail: an empty `needle`\n\nsits inside every body, so it passes silently and the gap reads `1.0000x`\n\n. The tool fails closed on evidence it `http_body_contains(\"ACK\")`\n\nconfirms the body said ACK. It does not confirm the ACK was for the right thing. Contract-level witnessing catches silent `hash_match`\n\nwith no `observed`\n\nfield) exits 2, not 0. You cannot trust a bill computed from evidence you cannot evaluate, so the tool refuses to compute one. I checked that by hand: malformed input exits 2, and zero verified successes with nonzero spend blocks rather than dividing by zero.If you log agent tasks at all, you already have `M`\n\n. What you probably do not have is `M'`\n\n, because most stacks never record an independent witness next to the exit code. That is the actual gap, and it is cheaper to close than it sounds: pick the one artifact each task is supposed to produce, record whether it is really there, and divide by that count instead. Everything here is offline, keyless, stdlib-only Python 3.13.5. It drops into CI or a pre-deploy hook with no daemon and no account, and it prints its own sha256 on the last line so you can pin the version you ran.\n\nHere is the part I have not settled, and I want your take. I set the masking tolerance at `1.20x`\n\nand I am not sure that is right for anyone but this fixture. Too tight and it fires on the honest 5% agent; too loose and it waves through a batch where a third of the spend bought nothing. And the deeper question is upstream of the threshold: what is your witness? If you run agents in production, tell me the single check you trust to confirm a task actually happened, the one that is not just the agent saying `ok: true`\n\na second time. I read every comment, and I am collecting the good ones.\n\nIf this was useful, the [200-and-lies witness gate](https://finops.spinov.online/blog/your-agent-returns-200-and-lies/) is where the honest denominator comes from, the [loop cost forecaster](https://finops.spinov.online/blog/loop-cost-forecaster/) is the numerator side, and the [pre-execution gate](https://finops.spinov.online/blog/pre-execution-gate-for-ai-agents/) is the pattern that stops the bad number before it spends. Follow along; the next one keeps walking down this stack.", "url": "https://wpnews.pro/news/cost-per-verified-success-your-exit-0-denominator-lies", "canonical_source": "https://dev.to/alex_spinov/cost-per-verified-success-your-exit-0-denominator-lies-5e6j", "published_at": "2026-07-25 03:51:39+00:00", "updated_at": "2026-07-25 04:28:12.003321+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/cost-per-verified-success-your-exit-0-denominator-lies", "markdown": "https://wpnews.pro/news/cost-per-verified-success-your-exit-0-denominator-lies.md", "text": "https://wpnews.pro/news/cost-per-verified-success-your-exit-0-denominator-lies.txt", "jsonld": "https://wpnews.pro/news/cost-per-verified-success-your-exit-0-denominator-lies.jsonld"}}