Danish multinational pharmaceutical Novo Nordisk is very interested in speeding up the time it takes to get drugs to market as patents expire. “If you have a blockbuster drug, a one-week delay can be $10 to $100 million,” says Stephanie Bova, the company’s digital transformation officer. “It’s massive money because you have less time on patent.”
Gen AI offered the possibility of dramatically speeding up multiple steps in the drug development process. And since Novo Nordisk was already carefully tracking how long its key processes took, it had an advantage that many companies didn’t. So it should’ve been relatively simple to sprinkle in some gen AI, see productivity improve, and watch the money roll in. But it wasn’t that easy. A drug development process has many parts, happening at different times in different departments.
“People are experts in their own domains but don’t necessarily know the next domain and how it all fits together,” Bova says. “The system is so big and complex that you’re not able to see all the performance at once.”
Process documentation might not match what people actually do in practice, and different people might do the same task in different ways. And some crucial tasks might be nearly invisible from the outside. The manufacturing team, for example, might sit in a completely different group and not be aware the drug is getting ready for FDA submission, and don’t have all their documents ready yet.
“So you’ve run very fast only to have to wait for them to catch up,” Bova adds.
This is just one of many challenges companies face when trying to measure the results of AI projects, and why surveys are so contradictory.
Looking at individual tasks, Novo Nordisk can show productivity improvements and clear positive benefits to its use of AI. But stepping back and looking at the company’s bottom line, the picture gets murkier. First, if critical steps are missed, then time to market won’t improve. It also takes years for a new drug to get to customers, so any positive bottom-line effects won’t be felt for a while. And that’s just the start of the ROI measurement problem.
To address its process blind spots, Novo Nordisk turned to the new generation of process mining: AI-powered real-time digital twins of operations.
“We partnered with process intelligence company Celonis to get a digital twin of our process data,” Bova says. “We were the first in the industry to apply it to the clinical setting.” The tool collects information from enterprise systems to track what employees actually do, rather than using surveys to collect information on what a fraction of employees remembered doing at some point.
The first project was a simple, seven-step process, and in creating a digital twin of it, Novo Nordisk discovered that, depending on who was doing it, it could be a five- or nine-step process. “If you get 10 different subject matter experts in a room, you get all kinds of interpretations, and you have drift over time,” she says.
The project exposed multiple flaws in existing processes. In some cases, employees needed to be retrained. In one, the user interface had to be updated. Once a process is standardized, though, there’s an opportunity to take the before picture, so there’s something to compare to afterward, to see if the AI augmentation or automation show any results.
Another thing they had to figure out ahead of time was decide what to do with any time savings that showed up.
“You don’t want to lay people off,” Bova says. “These are highly technical, hard-to-find talent. Maybe we want to think about redistributing teams a bit.”
Today, the company has several hundred AI agents in active deployment, tagged inside the digital twin infrastructure so they can be identified.
“If something screws up, we know exactly where to fix it,” she says, adding that the next phase is multi-agent orchestration. “Today, we have them connected, but we don’t have agents of agents.”
It’s too early to say if there’s ROI yet because, for drug development, the process takes years. “But by looking at the end-to-end process, my hope is we’ll find two years of cycle time to engineer out,” she says. “Two years quicker to market, compared to where we are now.”
Drugs that are already in the final phase of development won’t see as much acceleration, but those just starting out will benefit the most. The bottom line results, however, won’t show up for several years.
The pharmaceutical industry isn’t the only one where true value comes from optimizing multiple interconnected processes at once. According to PwC, tactical AI projects often don’t deliver measurable value, with tangible returns coming from enterprise-scale deployments consistent with business strategy.
In fact, many companies have seen neither increased revenue nor decreased costs from AI in the last 12 months despite nearly universal adoption of AI. Still, enterprise spending on AI is set to nearly double by the end of the year compared to last year, according to KPMG.
Most companies start on a smaller scale, rolling out AI chatbots to employees to help improve productivity. And the pace of adoption here has been staggeringly high, matched only by a lack of ability to measure the productivity gains that are supposed to be achieved.
Having a baseline is key, says Anand Rao, professor of AI at Carnegie Mellon University, but it’s difficult to measure in some cases, and all but impossible in others. Take for example insurance decisions where results can take years to show up. With life insurance, it could be decades, he says. And for some types of decisions, companies don’t have any measurements at all.
“There’s a social stigma to saying that I’m trying to look at your decision-making and how well you’re making the decisions,” he says. “As humans, we don’t like to be measured for our decisions.”
Then, when a decision turns out well in the end, people are happy to take credit. “If the decision goes badly, it’s something outside,” he says.
But even for specific tasks where measurement is possible, companies often don’t put in the work to make the measurements prior to rolling out AI tools. “We didn’t start with a baseline,” says Julie Averill, former EVP and global CIO of fashion retailer Lululemon. Averill is now CEO at Gold Thread, a digital transformation consultancy.
“We started with the assumption that AI was going to help people make better decisions,” she says. “And that sets you up to not being able to measure well.”
There are alternative metrics that a company can look at instead, she adds, like usage rates or user satisfaction. “This is happening, and it’s bringing benefits,” she says, “some of which you can see, and some you can’t. You have to trust the process. It’s just like the cloud. You know it’s the way of the future and you can see the benefits, but it’s hard to get there, and there’s a lot of change required. But the sooner you do that, the sooner you’re in the new way of operating and can really take advantage of it.”
There are other areas where hard metrics are more readily available, like customer service. “These are repeatable tasks, and it’s usually the first place companies automate with AI,” Averill says. “There are very tangible results you can measure, and you can have a very good baseline.”
Lululemon has also been using AI for years for better personalization and recommendations, and that’s also an area that can be quantified. And automation can reduce manual data entry, reducing error rates. AI can also be used to help with compliance monitoring, fraud detection, and predictive maintenance for equipment, which are all use cases that can be quantified.
But employee productivity in general? That’s a tough one to measure, and not just for Lululemon. One obvious way might be to look at layoffs in professions exposed to AI. After all, the headlines are everywhere. But in a report released in March, Anthropic found no signs of an increase in unemployment in highly exposed professions, those in which people are most likely to be laid off due to AI.
In early 2025, research firm METR attempted to quantify developer productivity by comparing how fast experienced developers were able to achieve tasks with AI and without. The result? Developers said they were expecting AI to speed them up by 24%, and estimated that AI had actually sped them up by 20%. But the data showed an altogether different story. Their use of AI actually slowed them down by 19%.
Of course, AI tools are getting better. METR attempted to do a follow-up study, again tracking tasks done with and without AI, but they couldn’t find enough developers willing to go back to the no-AI approach, even though the researchers were paying them to participate in the study.
There are anecdotal reports of companies where one engineer does the work of a hundred by using AI. Or that time the entire half-million-line Claude Code codebase was accidentally leaked and Korean developer Sigrid Jin created a clean-room rebuild in two hours, which he then pushed to GitHub, where it became the fastest project in history to hit 100,000 stars.
But as with anything else having to do with AI, the real picture is more complicated. With software development in particular, typing the code is actually just a fraction of what’s involved in developing software.
Research firm DX recently analyzed key engineering metrics from 400 companies, and in a recent report found that AI usage increased by 65% since November 2024, but AI-related productivity was just under 10%.
Just as it’s difficult to measure the productivity benefits of AI, it can also be tricky to measure the costs. When a company first starts using AI, costs might be relatively simple to estimate. What’s the total monthly subscription charges for the AI chatbots that employees are using? What’s the cost of training or fine-tuning a custom model? But when you move on to more complex use cases, the calculations get more difficult, says Averill.
“Now there are all the systems around the AI,” she says. “Those are harder to measure, but the impact is bigger.”
For example, if AI is embedded into business processes using RAG, there’s the ongoing expense of the API calls, but also the changes that need to be made to other systems, she says. And it just keeps getting more complicated every day. “We haven’t taken a very concerted effort to putting telemetry and instrumentation in place,” says Swaminathan Chandrasekaran, global head of AI and data labs at KPMG. He says that getting a comprehensive picture of the total costs of AI in an enterprise is like predicting the weather.
“The reason we have a pretty awesome weather prediction system in this country is because we have tens of thousands of weather stations that aggregate data,” he says. “Without that, we wouldn’t know the weather.”
Companies need to set up instrumentation to measure all the aspects of AI-related consumption, he says, starting with the number of tokens used, who’s using them, and how it correlates to work output.
“That measurement is fundamentally lacking,” he says.
At least when humans are using AI chatbots, there’s a limit to how many questions they’re physically able to ask, combined with predictable subscription costs. And when business processes are AI-enabled via RAG, the API calls to LLMs are being made by predictable, traditionally-scripted business systems.
But now, agentic AI is making everything worse because the agents can act unpredictably, and the number of API calls can quickly spiral out of control. In a report by the Boston Consulting Group, two-thirds of companies are reporting uncontrollable AI scaling expenses.
Another cost some companies might not anticipate well, or not track because it’s part of a different budget, is data-related cost. Whether preparing data for training or fine-tuning, using RAG embeddings, or setting up direct MCP access via agents, these costs can quickly add up when AI comes into the picture.
“Egress fees are one of the big ones,” says Tom Coughlin, IEEE fellow and president of consulting firm Coughlin Associates. “If you have to bring data out of the cloud, those egress fees could be considerable.”
Then there are all the human costs of deploying AI, he adds.
“There’ll be a lot of value that people get out of AI in the long run, but they need to know how to use it properly,” he says. “If they don’t have those skills, you’ll be at a disadvantage.”
Then there’s fixing problems. A majority of companies have had at least one AI-related incident in the last 18 months, with most resulting in financial loss, some over $500,000. Then there’s the AI that’s being embedded in everything.
“We know our direct costs,” says Andrew Johnson, CIO at Brownstein Hyatt Farber Schreck, a leading national law firm. “But where it becomes more difficult to measure is with platforms we already have in place, and SaaS applications that didn’t have AI capabilities,” he says. “They’re asking for extraordinary increases and attribute them to new capabilities due to AI. How much should be ascribed to AI? That’s a little wishy-washy.”
Even when AI saves money, there are often extra costs associated with that. For example, the firm was spending about $70,000 a year on a contract management platform. Building their own version with AI took about $40,000 in labor costs and another $3,000 a year for hosting. Ongoing maintenance will be minor for that particular application, he adds, totaling another couple of thousand a year.
But there are also other indirect costs that come with running your own applications, including security audits, vulnerability assessments, penetration tests, and code review.
“The more complex and riskier the platform, the less appetite there is for trying to create an in-house solution,” he says.
Still, the software development team is now dramatically more productive as a result of AI, with four or five developers able to do the work of 20 or 30.
But the productivity improvements don’t translate to labor savings, since there’s plenty of new work for the developers to do. “We have an enormous backlog of opportunities to develop solutions,” he says.
The tendency of work to expand to fill the time available isn’t just true for software development, says Carnegie Mellon’s Rao.
Say for example, AI is expected to lead to a 20% improvement in productivity, he says. “There were a hundred people doing it, and now we only need 80.” But at the end of the year, headcount hasn’t changed. “The tasks they were doing, there’s improvement,” he adds “But humans will add tasks to supplement or complement that 20%. It’s not that they’re going home an hour early, but they’re finding other value-generating activities.”
In fact, in some cases, increased productivity at a company can actually hurt the bottom line. Lawyers, for example, bill by the hour.
“Efficiency runs counter to our traditional ways of making money,” says Brownstein’s Johnson. “We have to think past that. It’s not detrimental to our long-term interest, but it’s a challenge in the short term. If we don’t do this, though, it’s likely we won’t be competitive in the mid- to long-term.”
So if a new AI tool helps an attorney with due diligence, there’s no straight line between the investment in that tool and increased revenues.
“It’s a given that it’s directionally right,” Johnson says. “But we can’t say it’s going to lead to a particular return.”