Here’s a question worth more than most of the AI hype takes clogging your feed: what happens to your velocity the day your frontier provider changes the deal? Raises the price. Deprecates the model you tuned your whole workflow around. Throttles you at the worst possible moment. Or just decides your use case, your industry, or your country isn’t one they want to serve anymore. If your honest answer is “I’d be dead in the water,” then you don’t have a strategy — you have a dependency. And it’s time to look hard at running some inference on your own metal.
I want to be careful here, because this is exactly the kind of topic where people reach for a dumb binary. Cloud vs. local. Frontier vs. open weights. Pick a team, plant a flag, go argue on the internet. Ugh, no. That’s not the argument I’m making.
You don't buy a lifeboat because you hate the ship. You buy it because the ocean doesn't care about your opinion of the ship.
I use the frontier models every single day. Claude Code is my daily driver and I’m not giving it up. The best hosted models are genuinely better than anything I can run at home, and they’re better by a lot, and today they’re cheap per token relative to what they do. All of that is true.
AND — both can be true — betting your entire ability to ship on a handful of providers you don’t control is a risk you’re taking whether you’ve named it or not. I’d rather name it. So let’s name it.
Reason One: The Token Bill Is a Business Model, Not a Law of Physics #
I’ve written before that tokens should be NRE, not COGS — that the expensive move is paying to infer the same thing over and over in production when you could infer once and export. Go read that one if you haven’t; I’m not going to repeat the whole argument here. But there’s a cousin of that idea that lives on the development side, not the production side.
When you’re running agents hard — and if you’re doing real agentic coding, you are running them hard — token consumption stops being a rounding error. It becomes a line item. A big one. And here’s the thing about a per-token opex line: it scales with your success. The more useful the agents get, the more you use them, the bigger the bill. That’s a business model designed to grow with you, which is great for the provider and quietly terrifying for your CFO.
Capex on a local box behaves differently. You pay once. The marginal cost of the ten-thousandth token is basically the electricity. For the right workloads — bulk classification, batch refactors, embeddings, the endless grinding background jobs that don’t need a genius, just a competent worker who never sleeps — moving that load onto metal you own can change the shape of the curve entirely. Flat line instead of a climbing one.
Per-token opex is a line that only goes up and to the right. Capex is flat after the buy. Somewhere there's a crossover, and it's closer than you think for high-volume, non-frontier work.
I am not telling you to run your coding agent’s frontier reasoning on a box in your closet. You can’t, not at parity, not yet. I’m telling you that a huge fraction of the tokens flowing through a mature AI workflow are doing unglamorous work that a good open-weights model on your own hardware does just fine — and every one of those you pull back in-house is a token that no longer shows up on someone else’s invoice with a markup on top.
Reason Two: Every Prompt Is Data Leaving the Building #
This one is closer to my heart, and regular readers know why. I don’t trust the “you are the product” model. I’ve deleted accounts over it. So it should surprise no one that I get twitchy about the volume and nature of what we casually pipe out to third-party inference endpoints.
Think about what actually goes into those prompts when you’re coding with an agent. Your source. Your architecture. Your unreleased features. Your customer schemas. The exact shape of the thing you haven’t announced yet. In a regulated shop, potentially PII, PHI, or data that lives under a residency requirement that says in plain language: this does not leave these borders. We send all of it out, constantly, because it’s frictionless and the tooling makes it invisible.
Now — the reputable providers have real enterprise terms. No-training guarantees, zero-retention options, the works. I’m not accusing anyone of reading your code for fun. But “I trust their terms of service” is a different security posture than “the data physically never left my network,” and if you can’t tell the difference, the compliance officer at a bank or a hospital certainly can. For some workloads the only acceptable answer is: it runs here, on hardware we own, air-gapped or damn close to it. That’s not paranoia. That’s the requirement written on the contract.
Local inference is how you say yes to those workloads instead of no. It’s the difference between “we can’t touch that data with AI” and “we run a private model against it and nothing crosses the boundary.” That capability has a dollar value the moment one deal depends on it.
Reason Three: Access Is Not Guaranteed, and Models Don’t Live Forever #
“Today’s models are the worst you’ll ever use.” I say it a lot, and I believe it — the trajectory is up and to the right and it’s not slowing down (well, maybe it is). But there’s a sharp edge on that optimism that people don’t like to look at: the specific model you built your workflow around is going to get retired. That’s the flip side of constant improvement. The model you fine-tuned your prompts against, the one whose quirks you learned, the one your evals are calibrated to — one day it’s deprecated with a sunset date, and you’re migrating whether you’re ready or not.
That’s the friendly version of losing access. The unfriendly versions:
- Price changes. The economics that make frontier tokens cheap today are subsidized by a land grab. Land grabs end.
- Rate limits and capacity crunches. You’ve felt this. Everybody’s slammed, your throughput craters, and your “agentic workflow” is now a very expensive way to watch a spinner.
- Policy and geography. Terms shift. Whole categories of use get restricted. Export rules and regional availability are set by people who have never heard of you and don’t care that your Tuesday depends on them.
- The provider just… changes. Gets acquired. Pivots. Decides your segment isn’t worth serving. This is not hypothetical in tech. It’s Tuesday.
Weights you’ve downloaded and hardware you own can’t be deprecated out from under you. Nobody sends your GPU a sunset email. That’s the entire point. It’s not that the local model is better — it’s that it’s yours, and yours doesn’t leave when someone else’s quarterly numbers say it should.
Reason Four: The Hedge — Or, Luck Favors the Ready #
Now the part that actually motivated me to write this, and the part I want to be honest about, because being honest beats being polished.
There’s a scenario where the frontier providers stagnate. Where the curve flattens, the magic slows, the improvements get incremental and the prices creep up because the growth story needs to become a profit story. In that world, the shop that spent some capital building real local capability — that learned to run models on its own metal, that has the muscle memory and the pipelines and the hardware already racked — keeps a big chunk of its velocity while everyone still renting by the token gets squeezed.
Do I think that stagnation scenario is likely? Honestly, no. There is an absolutely staggering amount of money at risk, and money that large does not sit still and let the party end. The competition is ferocious, the capital is (nearly) bottomless, and the incentives all point at keep making it better, keep making it cheaper. My actual bet is that the frontier keeps ripping.
But here’s the thing about a hedge: you don’t buy it because you’re sure the bad thing happens. You buy it because you can’t be sure it doesn’t, and because the cost of being ready is small next to the cost of being caught flat. This is Rickover-brained thinking and I come by it honestly — the Navy nuclear program didn’t drill casualties because it expected the reactor to fail. It drilled because the price of being unready for the one time it does is unacceptable. You train for the bad day on the good days. That’s the whole discipline.
Luck favors the ready. Spending some capital on local inference now — even if the frontier never stumbles — buys you three things that are worth having regardless of how the bet resolves: a cheaper token curve on your grunt work, a home for the data that can’t leave, and an insurance policy against the day the deal changes. If stagnation never comes, you’ve still got all three. If it does come, you’ve got a lifeboat while everyone else is reading the fine print on their SLA.
That’s not a bad trade in any future.
So What Do You Actually Do About It? #
You can’t learn to swim from a book. You definitely can’t learn it by having opinions about swimming on LinkedIn. So — go get a little wet.
I’ve been dorking around with exactly this. I set up Qwen3 on a GX10 and I’ve benchmarked local coding agents against the frontier on real hardware instead of guessing. I run models with thinking on and off and actually measured what it costs you rather than repeating what someone said on a podcast. None of it replaced my daily frontier driver. All of it taught me where the line actually is between “run this at home” and “pay the pros” — and that line is not where the loud people on either side think it is.
Start small and start real:
- Pick one high-volume, low-glamour workload — classification, embeddings, a batch job — and move it to open weights on hardware you control. Measure the bill before and after. Data, not vibes.
- Find the one dataset in your shop that can’t leave the building and prove to yourself you can run a useful model against it locally. That’s not a science project; that’s a capability your competitors may not have.
- Get the muscle memory now, while it’s cheap and low-stakes, so that if you ever need it in anger, it’s a Tuesday and not a fire drill.
You’ll keep using the frontier. I sure will. But you’ll be doing it as a choice instead of a dependency, and you’ll have a floor under you that you built yourself. In a market moving this fast, with this much money sloshing around and this many ways the deal can change, an owned floor is worth more than another clever prompt.
Buy some metal. Get it running. Luck favors the ready.
Conclusion #
Local inference isn’t about hating the cloud or pretending your closet can out-think a frontier lab. It can’t, and I’m not pretending. It’s about optionality: a cheaper curve on the grunt work, a lockbox for the data that can’t leave, insurance against losing access, and a hedge — a cheap one — against a stagnation I don’t even expect. Both can be true: the frontier is the best tool going and you should own enough of your own inference to survive the day the terms change. Have strong opinions, loosely held — but have a lifeboat.
If this got you thinking, or if you think I’m dead wrong and local is a boondoggle, drop me a note on LinkedIn. I’d genuinely like to hear where you’ve landed the run-local-vs-rent-tokens line in your own shop. And yes — if it helped, pay it forward.