The call usually opens with praise. The AI feature shipped on time, users love it and engagement charts are pointing the right way. Then finance closes the quarter, and the feature everyone celebrates loses money on every single request. That is the part the CTO called about. I get some version of this call every week. I work in developer relations at a GPU cloud provider in Silicon Valley, putting me in the room, or at least on the video call, when engineering teams decide how to buy and run AI infrastructure. The longer I do this work, the more familiar the pattern becomes. I watched companies learn cloud-cost discipline in the 2010s, usually after an end-of-month bill delivered a nasty surprise. GPU spending is the same lesson with two important changes: the hardware costs roughly ten times more per hour, and mistakes pile up faster. We’ve seen this movie.
Before moving into AI infrastructure, I spent years in data analytics at an automotive software company. One part of that job was cleaning up a decade of accumulated cloud enthusiasm, which sounds harmless until you inherit the bill. After we consolidated three overlapping analytics platforms into one, we cut about 220,000 dollars a year while keeping every capability intact. That money accumulated through reasonable-sounding subscriptions, one after another, because nobody owned the basic question: what did it cost to produce those numbers? The industry still hasn’t solved it. Flexera’s annual State of the Cloud research has for years found that organizations estimate more than a quarter of their cloud spend is wasted. An entire discipline, backed by the FinOps Foundation, grew around squeezing that waste back into a manageable shape. It took most companies years to learn those habits.
What bothers me is simpler: I keep seeing solid engineering teams drop that discipline the moment the purchase order says GPU. AI spend gets treated like a bold bet instead of an operating cost, and then the ordinary scrutiny disappears. That is where the trouble starts. The waste patterns of 2015 come back wearing 2026 pricing. The invoice tells you what you paid, separate from what you earned.
GPU capacity is priced by the hour, so teams naturally budget and report by the hour. It feels neat. It lines up with the bill. And it hides the problem that really crushes margins. The number that decides whether an AI feature survives is cost per request: everything you spend on inference infrastructure divided by the requests you serve. Those two metrics line up only when your hardware stays busy. For user-facing AI, that stays rare. Traffic moves with human attention, so it flares for a few hours and then drops off a cliff.
One team I worked with had reserved a cluster built for a peak that showed up for about two hours a day. On the invoice, the hourly rate looked almost cheap. Once we divided it by served requests, it was ugly, and the team had honestly seen it for the first time when we ran the numbers together on a call.
That division is the most useful exercise I can offer a reader of this column. Take last month’s total inference spend. Divide it by the number of requests you served. If the answer makes someone in the room go quiet, you have found money and you found it with arithmetic a spreadsheet has been waiting to do for you.
When the number looks ugly, the instinct is to push for a better rate or go hunting for another provider. I sell GPU capacity for a living, so I’ll say it plainly: the rate is rarely the issue. The unit price of AI compute keeps falling; Stanford’s AI Index has documented inference prices dropping by orders of magnitude in just a few years. That still leaves a team paying for capacity it barely touches. Waste eats the discount whole.
The fix lasts longer when you match the buying model to the workload itself, which is a point Andreessen Horowitz made well in its guide to the cost of AI compute: access to compute matters less than the shape of the commitment you sign for it. AI workloads usually split into two very different cases, and they want opposite deals. Sustained work, such as training runs, fine-tuning and batch processing, keeps hardware busy around the clock. This is what reserved or dedicated capacity is for. The economics are simply better when the machines stay hot. Reserved or dedicated capacity is made for that, and the per-unit economics pay you back for the commitment. Spiky work, which covers almost everything with a person on the other end, is the reverse. Usage-based pricing earns its markup there, because you only pay when you serve. The per-unit price goes up and the total bill drops. Finance teams resist that sentence until the numbers hit their own sheet.
The best production setups I see are hybrids. A team keeps a modest baseline, sized to the floor of traffic, the level demand almost never sinks below, and lets usage-based capacity soak up the rest. Teams under roughly ten million tokens a month often skip infrastructure entirely and stay on a model-as-a-service API until volume justifies the switch. The reserved slice stays busy. The bursts stay covered. The architecture quietly records a choice the team meant to make, which is rarer than it should be.
When a team asks me to review a GPU commitment, I keep coming back to the same three questions, and I would rather they ask them before the signature than after it.
Outside my day job, I’ve judged more than eight AI hackathons this past year, at Microsoft offices in Chicago and Mountain View, plus events with OpenAI and Google Developers Group. Even there, surrounded by teams building through a weekend, I can see the production problem waiting ahead: brilliant models, minimal thought about what serving them will cost. Nobody wins a hackathon with a unit economics slide. Plenty of companies quietly fail without one.
Years in data analytics left me with a conviction I repeat to every team willing to listen. A dashboard nobody costs out is a liability; an AI feature carries the same risk. The companies that survive the next pricing cycle will be the teams able to name their cost per request from memory and explain their infrastructure in one sentence, with a week of traffic data behind it, rather than those squeezing the lowest hourly rate from a vendor. Ten years ago, cloud bills taught engineering leaders to ask what their systems cost. Now the GPU bill is asking again, at ten times the stakes. The lesson lands harsher now: guessing survives only until the next ugly bill arrives at the worst time. The teams that move first will claim the margin everyone else is still chasing.