← All Blogs An interesting problem that we're currently trying to answer at Weave is "how can we quantify engineering?" An important metric we have here at Weave is "code output," where one unit of output is measured as one hour of an expert engineer's time pre-AI. But that metric doesn't really help us understand that code's impact or actual utility. So as a subset of quantifying engineering, we're trying to quantify impact.
What is impact even? #
The way I would define impact is: how does this change affect the systems and users that interact with it.
Something with positive impact could be refactoring duplicated logic across a shared pattern into a central abstraction, so other people can call this class instead of copying the same logic over. That would save a lot of time for others. Something with negative impact could be a change that exposes important secrets or introduces an anti-pattern.
A positive-impact change can introduce debt as well. Let's say a customer really wants a feature and the POC depends on it. You hack away at it and secure the deal. There are actually multiple legs still: how durable it is, how well utilized it is, and counterfactually what it brought now. So there's definitely a lot of dimensions to this.
Can a market price it? #
A little background on me: I really like markets. I've recently been doing a lot of research into prediction markets on the side and trade a small amount of volume on Kalshi (~$1 million in the past 30 days). So one of the ways I thought about approaching this problem was: can agents interacting on an exchange perhaps price a PR's impact or utility? The thesis of a market is to price an underlying good via what people are willing to buy and sell at. Maybe agents can do this with a PR.
Setting up the experiment was a little tough. Fleshing out the design is a bit non-trivial. What even is a PR's cash flow or settlement? What data can we feed these agents so they get some dollar value, and how do we even determine when it ends? To be frank I don't really have the most solid idea of that even now.
The main emergence I want to see is: can a market of different agent opinions aggregate qualitative reads on a PR to a stable, discriminating number? If I run the same PR through the exchange multiple times, is the price similar each time? When these agents disagree, why do they disagree?
One way to ground the dollar number: given that we have a real denominator from our code output metric, which is an expert engineer's time, the market could be pricing a multiplier. Let's say one unit of output is worth $100 of expert-engineer time. If the market converges on $3,000 for PR X and we know its output was 20 (so ~$2,000 of input), we can say the PR converted its input into 1.5x its cost in durable utility, inclusive of what it unblocked. Conversely, a PR with the same 20 output that converges at $300 converted only 0.15x — a spend that mostly didn't become lasting value. This is a very naive idea at the moment, but it gives us some fuzzy shape.
How the exchange works #
We have a central limit order book (CLOB): a double-sided auction where someone can say I'll sell you PR X at $520 and someone else can say I'll buy at $500. The players in this world can either place a make — putting out a flyer for your price for buying or selling this PR — or a take: instead of putting out your own flyer and waiting for someone, you trade against someone else immediately.
Notice the gap in the example: the best buyer says $500, the best seller says $520, so nothing happens — they're $20 apart. Trades only occur when someone gives in and crosses that gap. Makers earn money from that gap (buy low at their flyer, sell high at their other flyer) but risk getting caught with a stale price when new information shows up. Takers pay a little extra for trading right now.
Each agent has its own persona and its own personal theoretical value (theo), so we see how these different approaches blend and how they put their money where their mouth is. There is a central exchange tape that pushes updates into each agent's private research session, so its beliefs update on its own information, not just on what the market is doing. They can call a tool to inspect the current order book, a tool to place, and a bunch of research tools clamped to that exact merge time so they get context without any forward-looking bias. There's also a research budget. On a big PR, nobody reads everything. Agents that spend their budget well genuinely know more than agents that don't — and that difference of information is what makes the trading interesting.
So when does it end? A run is ephemeral: the market spins up for one PR, runs until a fixed number of trades or a time limit, and the last volume-weighted traded price is the market's answer. Nothing persists between PRs. I'll definitely look into other approaches, but this is the current naive process. Another thing to point out: this is a beauty contest at the moment. A future version will have time-based labels and markouts given how the code has transformed and been used.
A live run #
This PR was a small 4-line change (plus some tests) to our UI at Weave, where there was a redundant icon on our data connection page. The market opened around $13 and spent the run grinding down to $3, where it converged (volume-weighted over the last stretch of trades: 418 trades across 1,286 turns before the time cap hit).
The shape of the chart is the story. The early prints are agents pricing the rough idea of a PR. Agents then completed their research, marked their theos down, and started selling into anyone still quoting double digits. Each leg down is roughly someone's beliefs updating and the optimists getting picked off. By the end the order book tells you who believes what: people lining up to sell at $4–$7 and to buy at $1–$3, but the imbalance heavily lies on buying it for cheaper.
Is the price right? #
There is no true oracle, but it does get a number and did what we wanted: converge with a coherent story. The code output for this PR was ~.51 and attributed cost was ~$5 of AI token cost via Weave's AI cost attribution. Given these numbers, no one would call this damaging or valuable — a neutral utility change.
I'm excited to iterate more on this, and to give these agents more tools to research how users interact with these changes. Maybe some data from Slack or PostHog on the people who actually touch the change, and they might price it higher. But the ability to put some number on an engineer's impact is a step in the direction of quantifying engineering.