This year, for the first time, the world will spend more money running AI systems than training them. Training them was a capital bet placed by a handful of companies. Running them is an operating expense, and it lands on everyone’s P&L, every day, for as long as the system is in use. The question for business leaders is no longer whether AI works. It is whether they can afford to run it at scale.
The response so far has been rationing. Pilots cost so little that nobody tracked them. Production costs real money. So, companies are capping tokens, putting spending meters on screens and reserving the most capable models for an approved few. I understand the instinct. I have long argued that the discipline is in picking three or four focused AI projects and funding them properly so they can scale, rather than running a hundred disparate experiments. But that is a decision about where to spend. Rationing is a decision to underfund what you have already chosen. Rationing the most productive technology you have ever put in your people’s hands is a slow way to lose.
Instead, winning the economics of enterprise AI is determined by three questions: What drives the cost? Who sets the price? And what are you actually spending?
First, what drives the cost? The answer is simple: engineering does. AI agents, like the ones writing software, read roughly 600 words for every one they write. And since nearly all of that reading is repetitive, companies are really paying to reread their own documents again and again. The commercial Internet had the same problem. It got affordable not because bandwidth got cheap, but because the industry stopped sending the same data across the network repeatedly. The same fix works for AI. In an open-source project IBM helped build with Red Hat and Google, the system keeps what the AI has already read and sends each request to the machine that already holds it. On our own systems, 98 percent of what the AI needed was already in the cache, so the chips spent their time on new work instead of repeating old work. With caching and other optimizations that let the chips work in parallel, the same hardware handled seven times more work. The software is open source, and any company can use it today. Caching is not the only optimization technique clever engineers will come up with. A collection of them, worked on continuously, is what will bend the AI cost curve.
Second, who sets the price? Every major AI supplier now offers a discount when a customer’s data is reread, and that is welcome. But a discount on someone else’s platform is a price they set and can revise, while the same efficiency built into infrastructure you operate is a cost structure you own. Most AI budgets assume that serious results require the newest model on the newest chips. Neither is true. Highly capable models are now available with open weights, and they can run on infrastructure you control. For a growing share of work, running them there is becoming essential to managing cost. In one recent test on our own systems, IBM ran an open-weight model to power 3,000 AI agents at once. In one of the tasks on that system, designing hardware for quantum error correction, the open model delivered a 52-fold design speedup for $26 in inference costs. A leading proprietary model produced the better design — a 204-fold speedup — for $299, or more than 10 times the cost even with discounts for cached data. The lesson is not that every workload belongs on open models. When you are exploring hundreds of early candidates, the $26 run is the right tool; when you are finalizing the chip, you pay the $299. You should decide which work runs where, and on whose terms. Choosing models that are fit for purpose is another way to bring costs down.
Third, what are you spending? Companies are used to governing technology against units they can count: a server, a license, a seat, a CPU hour. Even variable cloud spending has a legible chain: an application runs, infrastructure is consumed, and the bill rises when more customers show up. AI breaks that chain because it is a probabilistic machine. The same request might cost two cents one time and two dollars the next, because the system took a different route to the same answer. Run that without control, and no one in your company is deciding what the work costs. The machine is. That is why the systems built above the model matter: they make agents more reliable, limit the tools and data they can touch, and keep costs predictable. The goal is not the cheapest token. It is the highest token yield: the most value for every dollar spent.
All three questions land in the same place. Between your data and your models sits an AI operating layer that connects systems never built to speak to each other, governs which people and agents can use which models, and what each may see and do, decides where AI workloads run based on value and economics, and keeps the record a regulator will one day ask for. This layer is where IBM’s capabilities come together for our clients. We ensure that you own that layer, so you own the economics of AI.
The cost of AI is a consequence of how you deploy it and of where the value sits. A great deal of that value sits inside your own company. Roughly four-fifths of enterprise data is unstructured and largely untouched, and it is not on the open internet. It sits in the claims history of an insurer, the underwriting file of a bank, the patient record of a hospital, the telemetry of an aircraft fleet. This data is too large, too regulated and too embedded in systems that cannot to be shipped wherever a vendor prefers to process it. So the intelligence has to come to the data. That is as much a matter of sovereignty and security as of cost.
What you actually pay should be decided inside your own company. The frontier labs will keep driving down the cost of their models, but that only sets the floor. If your supplier built the AI operating layer, then it is not a control; it is a meter. Own that layer yourself, and you can afford to run AI at scale. You will be the one bending the AI cost curve.
The opinions expressed in Fortune.com commentary pieces are solely the views of their authors and do not necessarily reflect the opinions and beliefs of Fortune*.*
breaks the traditional barrier between audience and newsroom. The show transforms
Fortune Daily Fortune’s trusted reporting into actionable, conversational, and entertaining insights for an emerging class of business leaders.
Watch here.