Cost Per Verified Success: Your Exit-0 Denominator Lies

A developer warns that cost-per-task dashboards for AI agents are misleading because they count exit code 0 as success, even when the agent silently fails. The fix is a 'cost per verified success' metric that uses independent witness checks to confirm task completion, always yielding a higher true cost than naive calculations.

Your cost-per-task dashboard is doing division. Spend on top, "successful tasks" on the bottom. The number it prints is the average cost of one agent task getting done. Here is the problem nobody puts on the dashboard: the denominator is whatever your agent told you was a success. On most stacks that means exit code == 0 , or ok: true , or an HTTP 200. That is the actor grading its own homework. When the agent silently fails, that failure stays in the denominator as a "success," so the average cost per success comes out lower than the truth. Your cheapest-looking number is the one you can least trust. Cost per verified success is agent spend divided by witnessed successes, not by exit-0. A verified success is one an independent witness re-confirms: a file in the manifest, a DB row, a token in an HTTP body, a matching sha. Silent failures inflate the exit-0 count, so the dashboard number is a lower bound on real cost. AI disclosure.I wrote verified cost.py with an AI assistant and ran every case myself before publishing. Every terminal block below is pasted from a real run on Python 3.13.5, stdlib only. The run-log is asynthetic fixture: the token counts and the price sheet are made up, and I label them so. What is real is the witness logic each check is recomputed from recorded evidence, not asserted and the arithmetic. I have no production incident and no invoice to sell you here. bot2 is new and its lifetime spend is zero dollars. What I have is a script that runs and a number you can reproduce. Because "task" and "successful task" are two different measurements, and the cheap one is wearing the expensive one's name tag. exit code == 0 is cheap: it is the process telling you it thinks it finished. It is a self-report from the same actor whose work you are trying to price. On this blog the shape keeps recurring https://finops.spinov.online/blog/your-agent-returns-200-and-lies/ : a tool returns 200 and lies, a green check reconciles against nothing, an approval is not immutability. Cost inherits the same disease. If your success count is self-reported, your cost-per-success is self-reported too, and it always rounds in the flattering direction. Watch it happen on one line of a run-log. An agent calls a "create order" endpoint. The endpoint returns HTTP/1.1 200 OK with a body of {"status":"RATE LIMITED","order":null} . The process exits 0, because 200 is not an error. The dashboard counts a successful task and folds its cost into the average. No order was created. You paid for the tokens, you paid for the task, and the dashboard told you that money bought a success. It did not. Multiply that by an overnight batch and your per-success number drifts away from reality while looking perfectly healthy. The fix is not a better dashboard. It is a better denominator. A verified success is a task that 1 exited 0 and 2 has its effect re-confirmed by a check independent of the agent. That "and" matters. It makes verified successes a subset of exit-0 successes, which is the whole reason the math has a direction. Call M the count of exit-0 tasks and M' the count of verified ones. Because every verified success is also an exit-0 success, M' <= M for any log you will ever feed it. So: naive cost per success = total spend / M verified cost per success = total spend / M' M' <= M, so this is = naive understatement factor = verified / naive = M / M' = 1, always That inequality is not a finding. It is arithmetic. naive cost <= verified cost for every possible run-log, with equality only when M' == M , meaning zero silent failures. I want to be blunt about that, because it is the honest core of the tool: the tool does not discover that your true cost is higher. It proves it is at least as high and then measures by how much. The direction is guaranteed. The magnitude is what you did not know. A witness has to be concrete or it is just vibes with a checkmark. verified cost.py ships four kinds, and each one is recomputed from recorded evidence rather than trusting a boolean somebody wrote down: python def eval witness w : kind, ev = w "kind" , w "evidence" if kind == "file exists": return ev "target" in ev "manifest" path actually present if kind == "db row present": return ev "target" in ev "rows" row id actually there if kind == "http body contains": return ev "needle" in ev "body" token actually in the body if kind == "hash match": return ev "expected" == ev "observed" sha actually matches raise ValueError "unknown witness kind" anything else - fail closed The real function has the type-checking and the fail-closed raises spelled out; this is the spine. The point is that a witness result is a function of evidence you can inspect, not a second self-report. If the create order body says RATE LIMITED , then http body contains "ORDER CONFIRMED" returns false no matter what the exit code claimed. You can paste your own bodies in and rerun. The check does not care about the agent's opinion. Here is the meter on a synthetic overnight batch of 12 tasks. Ten exited 0. Two failed honestly a migration that returned exit 1, a docker build that OOM-killed with 137 , and the dashboard already knows about those. The interesting three are the ones that exited 0 and did nothing: t03 got the RATE LIMITED body above, t04 produced an artifact whose sha did not match what was expected, t06 claimed to update user-90 but that row is not in the table. bash $ python3 verified cost.py report fixtures/runlog.json id spend exit0 witness verified t03 $0.3540 yes FAIL no t04 $0.5850 yes FAIL no t06 $0.2775 yes FAIL no ... total spend = $3.4665 M exit-0 successes = 10 <- the denominator your dashboard uses M' verified: exit0 AND witness = 7 silent failures exit0, witness FAIL = 3 't03', 't04', 't06' spend on silent failures = $1.2165 bought an exit 0, witness rejected it ------------------------------------------------------------------------------ naive cost per success = total/M = $3.4665 / 10 = $0.3466 verified cost per success = total/M' = $3.4665 / 7 = $0.4952 understatement factor = M/M' = 10/7 = 1.4286x arithmetic, not a measured effect Read the raw pieces, not just the ratio. M is 10, M' is 7, and I print both so the 1.4286x cannot hide anything. On this batch the dashboard would tell you each success cost 35 cents. The witness says 50. That is the same 1.4286x the report prints: the true per-success cost sits 43% above the dashboard's number equivalently, the dashboard reads 30% below the truth , and it is not a rounding error. $1.2165 of the $3.4665 you spent, better than a third of the bill, bought exit-0s that the witness threw out. The dashboard counted that third as wins. I want to be precise about what is real here and what is not. The 1.4286x is 10/7 , and the 10 and the 7 are counts I chose when I built the fixture. So the magnitude is synthetic. What is not synthetic is the direction and the mechanism: on any log, the moment one exit-0 task fails its witness, M' drops below M and your true cost climbs above what the dashboard shows. The tool computes M' by rerunning the checks, so on your own log the number is yours, not mine. A meter you read after the fact is a postmortem. The point of this franchise is to gate before the spend lands, the same way the pre-execution gate https://finops.spinov.online/blog/pre-execution-gate-for-ai-agents/ decides whether an action runs at all. So verified cost.py gate takes two policy knobs, a max verified cost per success and a max understatement factor, and returns a CI exit code: 0 to pass, 1 to block, 2 to fail closed on garbage input. bash $ python3 verified cost.py gate fixtures/runlog.json 0.15 1.20 naive=$0.3466/succ verified=$0.4952/succ understatement=1.4286x policy: max verified=$0.1500/succ max understatement=1.2000x BLOCK: verified cost $0.4952/succ exceeds budget $0.1500/succ BLOCK: understatement 1.4286x exceeds tolerance 1.2000x silent failures mask the bill VERDICT: BLOCK exit=1 Two independent reasons fired. The budget one is a generic FinOps cap. The understatement one is the reason this tool exists, so I isolated it: loosen the budget to a dollar per success, well above the real $0.4952 , and the gate still blocks. bash $ python3 verified cost.py gate fixtures/runlog.json 1.00 1.20 policy: max verified=$1.0000/succ max understatement=1.2000x BLOCK: understatement 1.4286x exceeds tolerance 1.2000x silent failures mask the bill VERDICT: BLOCK exit=1 That is the whole idea in one exit code. Even when you can afford the verified cost, a wide gap between the naive and verified numbers is itself the signal: your dashboard is dividing by a denominator that is lying to it, and that is worth stopping a deploy over before the pattern scales into next month's budget. A gate that always fires is a stuck alarm, not a control. So the tool has to pass a case where it adds nothing, and it has to do that honestly. If your agent never silently fails, then exit code == 0 really does mean success, M' equals M , and there is nothing to correct. Here is that case, five tasks, every exit-0 witnessed: bash $ python3 verified cost.py gate fixtures/honest zero.json 0.30 1.20 naive=$0.2244/succ verified=$0.2244/succ understatement=1.0000x VERDICT: PASS exit=0 Understatement 1.0000x . Naive equals verified to the cent. The masking check is silent and the gate passes. That is the falsifier working: if exit-0 never lied on your stack, this tool changes nothing and says so out loud. Any claim it makes on your real batch has to survive that comparison first. Now the reasonable objection: "fine, but our agents are healthy, we pass 95% of tasks, the gap must be negligible." It is not zero, and the arithmetic says exactly how not-zero. understatement = M/M' = 1/ 1-s where s is the silent-failure rate among exit-0 tasks. I swept s to show the magnitude at each rate this table is arithmetic, not a measurement of any workload, and it is labeled that way in the output : k M' s SE s M/M' 1/ 1-s 0 200 0.0000 0.0000 1.0000 1.0000 10 190 0.0500 0.0154 1.0526 1.0526 40 160 0.2000 0.0283 1.2500 1.2500 60 140 0.3000 0.0324 1.4286 1.4286 100 100 0.5000 0.0354 2.0000 2.0000 200 0 1.0000 0.0000 UNBOUNDED 1/0 <- M'=0, verified cost UNBOUNDED, gate BLOCKS At a 5% silent-failure rate, the "healthy 95% agent," your true cost is 1.0526x the dashboard number. Small, but not nothing, and it only grows: the lower your real success rate, the wider the gap, because you are dividing the same spend by an ever-smaller denominator. And the bottom row is the one I most wanted the tool to handle without crashing: when every exit-0 task is a silent failure, M' is zero, verified cost is unbounded, and the gate blocks rather than dividing by zero and printing a comfortable-looking number. One more honest note buried in that table. If you use exit code itself as the witness, M' equals M for every s , so the gap is 1.0 always. A dashboard is not doing bad arithmetic. It is using the null witness: the actor as its own judge. The gap this tool measures is precisely the information a real witness adds beyond the agent's self-report. No witness, no gap, no visibility. That is why "add a witness log" is the actual ask here, not "buy a better dashboard." I would rather you trust the small claim than oversell the big one. 1.4286x , $0.4952 , $1.2165 come from a fixture I built. They illustrate the arithmetic. They are not a measurement of any real agent fleet, mine or anyone's. The direction naive is a lower bound is general; the size is not. M' . If your only "independent" check is another call to the same flaky service, you have two self-reports, not a witness. The four kinds here file exists , db row present , http body contains , hash match are the ones I could make recompute from evidence with no network. Yours may need more. And the quiet version of a garbage witness is one too weak to ever fail: an empty needle sits inside every body, so it passes silently and the gap reads 1.0000x . The tool fails closed on evidence it http body contains "ACK" confirms the body said ACK. It does not confirm the ACK was for the right thing. Contract-level witnessing catches silent hash match with no observed field exits 2, not 0. You cannot trust a bill computed from evidence you cannot evaluate, so the tool refuses to compute one. I checked that by hand: malformed input exits 2, and zero verified successes with nonzero spend blocks rather than dividing by zero.If you log agent tasks at all, you already have M . What you probably do not have is M' , because most stacks never record an independent witness next to the exit code. That is the actual gap, and it is cheaper to close than it sounds: pick the one artifact each task is supposed to produce, record whether it is really there, and divide by that count instead. Everything here is offline, keyless, stdlib-only Python 3.13.5. It drops into CI or a pre-deploy hook with no daemon and no account, and it prints its own sha256 on the last line so you can pin the version you ran. Here is the part I have not settled, and I want your take. I set the masking tolerance at 1.20x and I am not sure that is right for anyone but this fixture. Too tight and it fires on the honest 5% agent; too loose and it waves through a batch where a third of the spend bought nothing. And the deeper question is upstream of the threshold: what is your witness? If you run agents in production, tell me the single check you trust to confirm a task actually happened, the one that is not just the agent saying ok: true a second time. I read every comment, and I am collecting the good ones. If this was useful, the 200-and-lies witness gate https://finops.spinov.online/blog/your-agent-returns-200-and-lies/ is where the honest denominator comes from, the loop cost forecaster https://finops.spinov.online/blog/loop-cost-forecaster/ is the numerator side, and the pre-execution gate https://finops.spinov.online/blog/pre-execution-gate-for-ai-agents/ is the pattern that stops the bad number before it spends. Follow along; the next one keeps walking down this stack.