AI adoption is not the same as delivering outcomes
“If there is something we learned with AI, it is that we don’t know what productivity in software is”. This was a quote I’ve heard from another engineering leader that resonated with me.
AI is reshaping how teams work in ways that were impossible to expect just a few years ago. The gap between idea and execution is smaller than ever. With that, it’s expected that we try to learn how to use the new tools in the best way possible.
That is what the software industry is doing. The tools are changing every day, the methods are changing every day. We are in the first year of this shift, and in the last 6 months we have already seen a lot. Agents running themselves in loops, spending on tokens as a strategy, then budgets to contain the spending, to cite some examples. And that doesn’t include another plethora of development techniques that have shown up along the way.
Some of these ideas were clearly flawed from the start. But still, we are trying to change how we work. Innovating and failing is part of it.
The problem is that we are experimenting with the method and not measuring the actual outcome.
We often measure intermediary results instead of the real goal
“Our teams are committing 90% of PRs with AI”. When industry leaders are touting their AI adoption as an achievement, they are highlighting how good they are at adopting a trend. What needs to be brought back to the conversation is value. AI is a tool, like many others, in the process of delivering customer value.
While the CEO’s example is an obvious one, the same challenge appears in many other areas. I was recently in a demo for an engineering metrics tool that helped analyze which AI prompts were leading to more PRs being merged. And my thought looking at it was probably the same as people listening to those CEOs: how does that connect to delivering more value? The problem here is also not that we are using more data to talk about how teams work. We need to become better at understanding team effectiveness and data is a part of it. That is an argument I completely stand behind. But we cannot go from vibes to false precision.
Show me the time (and money) #
The usual problem with this topic is that we end up circling toward the same question: what is value? If we go too broad, we cannot measure it, meaning we cannot assess if anything is working. However, elevating output metrics to a higher level is also not the solution. Metrics don’t work across all levels of an organization.
If you are the CEO of a company, token spend doesn’t mean much apart from the size of your AI bill. If you are a software engineer, you won’t be able to change the revenue predictions for the year. Depending where you are in your organization, you will have to find the right tradeoff between going as broad as possible, and having something that you can act on and understand the results. Metrics that are useful at one level of the organization won’t necessarily be useful at different levels.
If you are managing a software team (or group of teams), your answer is probably a mix of money and time to deliver a valuable customer result. In other words: investment and cycle time. The cycle time part of it is hopefully clear. If there is a perceived customer need, for example if they want to be able to find the right product faster, then what your team needs to optimize for is how fast it can deliver that result. How long from when the customer expressed their need, either directly or indirectly, to when you deliver the feature that enables it?
Using cycle time to measure software development brings multiple advantages. Firstly, it focuses on actual value that is hard to game through intermediary steps. Your customers don’t care how many tokens you spent or saved delivering the result, or if agents were sentient while building it. They care if their problems were solved and how quickly that happened. Admittedly, it is possible to game cycle time by reducing the size of each slice. But that’s a positive move as smaller batches will lead to better performance. A win-win situation, as they say. And cycle time also allows you to create intermediary metrics that use the same principle. The cycle time to value mentioned above includes the whole chain and also the ability of your company to perceive customers’ requirements. For an engineering team, it might be good enough to think about the time between requirements landing on your team (as a project, for example) and being delivered to customers. Within that, you can also think about the cycle time of each ticket, representing a slice of that value, or each PR, focusing on the day-to-day engineering work. It’s the same metric applied at different detail levels of the system, and they all have the same goal.
Different metrics are measuring the same concept at different levels: how long does it take to get a change done?
However, time is not everything, and it’s becoming more obvious as the technology spend shifts from only humans to humans and agents. For a long time tech companies operated on a version of a “build it and it will be worth it” principle. The idea is that if you can build something useful in software, it will scale enough to make the cost worthwhile.
That was never a great practice, even before AI, as many companies would struggle to scale. But it was often hidden behind many years of investment runway. You could see it in every company: a project gets delivered with great feedback and everyone gets a cake. It doesn’t matter that the project took three years with a cost of ten million dollars and will only make 100k in return.
As the investment capacity lowered and the AI bill got added to the costs, the question of materiality of an investment becomes even more critical. The question engineering leaders need to become better at answering is how much investment it will take to deliver some value. Investment that is composed of time, with the compensation of engineers and other team members, and tooling, with the cost of AI tokens.
And engineering leaders should move toward that question, not away from it. Focusing the conversation on investment and time forces us to ask what actually leads to faster, cheaper delivery. Which practices, which tools, which principles? In other words, it tells us how to deliver more value.