A dashboard shows a simple improvement:
| August | September | |
|---|---|---|
| Cost per successful outcome | $1.80 | $1.20 |
That's a 33% decrease.
At first glance, the story seems obvious: the economics improved.
But there are very different ways to produce exactly the same change.
Suppose the August metric came from $180 of attributable cost and 100 successful outcomes.
In September, one possible history looks like this:
attributable cost: $180 → $120
successful outcomes: 100 → 100
The system produced the same number of successful outcomes while attributable cost decreased.
But another history could look like this:
attributable cost: $180 → $180
successful outcomes: 100 → 150
Now cost didn't decrease at all. The system produced more successful outcomes from the same attributable cost.
Both histories produce the same dashboard:
$1.80 → $1.20
Same metric change. Different underlying story.
I've been thinking about this while exploring economic evidence for AI systems, and it exposed an assumption I hadn't paid enough attention to:
If I store an economic metric over time, I can explain how the economics changed.
I'm no longer convinced that's true.
Storing $1.80 for August and $1.20 for September preserves the observation. It tells us that something changed in the calculation.
It doesn't necessarily preserve enough information to explain what changed underneath it.
A value such as $1.20 per successful outcome is derived state.
Behind that number may be executions, retries, model and tool usage, provider costs, attribution decisions, successful outcomes, and the rules used to decide which costs and outcomes belonged in the calculation.
The final ratio compresses those facts into something useful.
But compression has a consequence: information disappears.
Imagine that all we preserve is something conceptually equivalent to:
period metric value
2026-08 cost_per_successful_outcome 1.80
2026-09 cost_per_successful_outcome 1.20
We can answer one question immediately:
What changed?
The metric decreased.
But the stored values alone cannot tell us whether execution became cheaper, more outcomes succeeded, provider rates changed, or the measurement itself changed.
The ratio is an output. It isn't an explanation.
That suggests a different engineering question.
Instead of asking only whether we can calculate an economic metric, we can ask whether we can reconstruct enough of what produced it to explain a change later.
Conceptually, I think of it more like this:
EXECUTION EVIDENCE
+
VALUATION EVIDENCE
+
OUTCOME EVIDENCE
+
MEASUREMENT DEFINITION
↓
ATTRIBUTION / AGGREGATION
↓
DERIVED ECONOMIC METRIC
The exact architecture can vary enormously. A small product doesn't need a data-lineage platform just to calculate its AI costs.
The important point is simpler:
A derived economic metric becomes much more useful when the evidence and rules that produced it remain inspectable.
Once those inputs disappear, we may still know that the economics changed while losing the ability to determine what kind of change we're looking at.
Suppose September's cost per successful outcome is lower than August's.
One possibility is that the runtime actually behaved differently. Maybe fewer retries occurred, a cheaper model handled more requests, or the workflow required fewer external tool calls.
Another possibility is that execution stayed almost identical, but the same work was valued differently because a provider rate changed.
Or perhaps neither execution nor valuation explains the movement. The system simply produced more outcomes that satisfied the success condition.
These are already different explanations for the same metric change.
But there is another case that I find more subtle: the system may not have changed in the way the metric suggests. The measurement may have changed.
Imagine that in August an outcome was considered successful when:
validation_passed = true
Then, in September, the product adopts a stricter rule:
user_accepted_result = true
The September metric may move even if the underlying execution behavior remains stable.
The denominator now means something different.
The same problem exists on the cost side.
Suppose August includes model and tool costs:
attributable_cost =
model_cost
+ tool_cost
Then September starts including an allocated share of infrastructure:
attributable_cost =
model_cost
+ tool_cost
+ allocated_infrastructure_cost
Again, the metric changes.
But interpreting that movement as a change in runtime economics alone would be misleading. Part of the difference comes from changing what the measurement includes.
That leaves us with two fundamentally different questions:
Did the system change?
and:
Did the way we measure the system change?
If we can't distinguish them later, historical comparison becomes much harder to interpret.
This is where storing only the final value starts to become uncomfortable.
Imagine seeing this six months later:
2026-08 cost_per_successful_outcome 1.80
2026-09 cost_per_successful_outcome 1.20
The metric name is identical.
But was the success condition identical?
Was the attribution boundary identical?
Were costs valued under the same rules?
Did both periods include the same population of executions?
Were they evaluated using equivalent observation windows?
If the answer to one of those questions is no, then comparing $1.80 and $1.20 may still be useful — but we need to understand what kind of comparison we're making.
This made me think differently about metric definitions.
A definition isn't necessarily static configuration surrounding the "real" data. Once historical economic values depend on that definition, the definition becomes part of the context required to interpret those values.
That doesn't mean every product needs a sophisticated policy-versioning system.
It does mean that if a success rule, attribution boundary, or valuation policy can change, there is an engineering question worth asking:
What needs identity or version history if we want to explain historical economic changes later?
At this point, the problem starts looking less like storing metrics and more like preserving relationships.
If a dashboard tells me:
cost_per_successful_outcome = $1.20
I'd like to be able to move backward from that number and ask which population of executions contributed to it, which costs were attributed to that population, which outcomes were counted as successful, which rules classified them that way, and which valuation context turned resource consumption into monetary values.
I don't necessarily need every answer to be perfectly precise.
Shared infrastructure may require allocation. Some provider costs may arrive late. Outcome evidence may still be pending. Certain values may be estimates.
Reconstructability doesn't eliminate uncertainty.
It makes that uncertainty inspectable.
This distinction matters because provenance should not be confused with proving a single root cause. If retry frequency increased while provider rates decreased, both may have contributed to the final movement.
The evidence may let us decompose the change without justifying a claim that one event "caused" the entire result.
So when I say that the number needs provenance, I don't mean that every startup needs a full data-lineage platform.
I mean something narrower:
If an economic number matters enough that we expect to explain its movement later, we should think about whether the evidence, definitions, and boundaries that produced it can still be identified.
Without that, a historical metric can remain numerically available while its explanation slowly disappears.
I've found it useful to investigate a change through four questions.
Did execution change? Did the runtime perform different work? Retries, model calls, tool usage, routing, or workflow behavior may have changed.
Did valuation change? Did the monetary interpretation of that work change? The same measured usage can produce different economics under different provider rates or valuation policies.
Did the outcome change? Did a different number of outcomes satisfy the relevant success condition? Did previously pending evidence become available?
Did the measurement change? Did the success definition, attribution boundary, included population, or observation window change?
I don't think of these as a universal taxonomy of economic change. They're investigative questions, and more than one answer can be true at the same time.
That's precisely why the final ratio is insufficient on its own.
A metric moving from $1.80 to $1.20 tells us where to start looking.
It doesn't tell us what we'll find.
There is another complication.
Sometimes the underlying execution doesn't change at all, but the metric does.
Suppose 100 AI research workflows completed yesterday. At the end of the day, the system had enough evidence to confirm 70 successful outcomes.
Today, additional outcome evidence arrives from another system. Fifteen previously unresolved outcomes can now be confirmed as successful.
No new workflow ran. No additional model call was made. No retry occurred. No provider rate changed.
Yet a metric based on confirmed successful outcomes can move because the denominator changed from 70 to 85.
This is not necessarily a correction of a bad calculation.
Yesterday's value may have been correct given the evidence available yesterday. Today's value may also be correct given the evidence available today.
The difference is what the system knew when each value was produced.
That creates an awkward question for historical economic metrics:
What does "the August value" mean?
Does it mean the value the system was able to calculate at the end of August?
Or does it mean the value we can reconstruct today using everything we now know about August?
Those are not always the same question.
Imagine that on August 31 the system had:
attributable cost: $180
confirmed successful outcomes: 70
Later evidence eventually raises the confirmed outcome count to 85.
A historical dashboard could preserve what was known at the time.
Or it could recalculate August using the evidence available now.
There are legitimate reasons for either view.
The important engineering problem is making clear which one we're looking at. Otherwise a historical metric can appear to have changed even though the underlying executions are already immutable history.
This is one reason temporal context matters.
For economic analysis, we may care not only about when the execution happened, but also when relevant outcome evidence became available and which observation window was used to produce a particular view.
I wouldn't turn every metric system into a temporal database because of this.
But if evidence can arrive late, silently overwriting a derived historical value can erase something useful: what the system was justified in concluding at an earlier point in time.
The opposite choice has a cost too. If we freeze every historical metric forever, later evidence can never improve our understanding of what actually happened.
So the question isn't simply whether historical metrics should be mutable or immutable.
It's whether the system can distinguish the perspectives it needs.
This is where I think the word "reconstructable" needs some care.
It would be easy to interpret it as:
Given the same execution, the system should always reproduce exactly the same economic number.
But that only works if all of the inputs and interpretations are fixed.
In real systems, they may not be.
Outcome evidence can arrive late. Cost evidence can be incomplete. Allocations can be estimated. A valuation policy can be corrected. A measurement definition can evolve.
So reconstructability is not necessarily about forcing one number to remain eternally unchanged.
It is about being able to understand which evidence and rules produced a particular answer.
That might let us ask:
What did we know then?
and separately:
What can we reconstruct now?
Both can be useful.
What becomes dangerous is having two different answers without being able to explain why they differ.
This changes how I think about the relationship between evidence and a dashboard.
A dashboard value is useful precisely because it compresses complexity. Nobody wants to inspect hundreds of executions, provider records, outcome events, and policy decisions every time they want to know whether economics improved.
But convenience shouldn't turn the compressed value into the only surviving truth.
Conceptually:
EXECUTION EVIDENCE
+
ECONOMIC / VALUATION EVIDENCE
+
OUTCOME EVIDENCE
+
DEFINITIONS AND BOUNDARIES
↓
ATTRIBUTION / AGGREGATION
↓
ECONOMIC VIEW
The value displayed on the dashboard is one result of that process.
That makes me increasingly interested in treating important economic metrics less like isolated facts and more like reproducible views over evidence.
Not perfectly reproducible in every case. Not free from estimates or uncertainty. And not necessarily implemented by recalculating everything from underlying events every time someone opens a dashboard.
The architectural point is about where authority lives.
If the only thing that survives is:
cost_per_successful_outcome = 1.20
then $1.20 eventually has to carry more meaning than it actually contains.
If the evidence and interpretation context survive too, the metric can remain what it was supposed to be: a useful derived answer.
I'm exploring this question while thinking about the economic evidence model behind Licenzy.
The more I work on it, the less interesting I find a final economic metric in isolation.
I still want the number.
But I also want to know whether I can move backward from that number to the execution evidence, economic evidence, outcome evidence, and definitions that made the number possible.
Not because every metric needs forensic-grade lineage. The architecture should be proportional to the questions the business actually needs to answer.
But once a number starts influencing decisions, historical comparisons, or claims about whether an AI workflow became more economically efficient, losing its provenance becomes more consequential.
This is also where I think economic observability becomes an interesting engineering problem.
Not as a standardized discipline, and not as a claim that economic questions are identical to technical observability. The useful similarity is narrower.
When latency changes, seeing the new latency value is often only the beginning of the investigation.
When an economic metric changes, I think the same instinct can be useful.
The change is an observation.
The explanation lives underneath it.
The Licenzy Guide What Did One Successful AI Outcome Actually Cost? explores the economic question behind cost per successful outcome: what belongs in the cost, what counts as a successful outcome, and what that ratio can and cannot tell us.
The engineering question I've been exploring here starts after that number exists:
If the metric changes, did we preserve enough evidence to explain what changed underneath it?
And that leaves me with a question I'm still thinking about:
Should an economic metric be treated as the source of truth — or as a reproducible view over the evidence that produced it?