# You're Not Buying Compute, You're Buying Utilization > Source: > Published: 2026-08-20 16:08:39+00:00 [AI](https://sourcefeed.dev/c/ai)Article # You're Not Buying Compute, You're Buying Utilization Every honest cost model of open-model inference reaches the same verdict, and the reason should change how you decide. [Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein) A developer on dev.to spent four months and a couple of rented A100s testing whether self-hosting open models still beats paying for API access. Their verdict: it doesn't, not until you're pushing serious sustained volume — they cut their own bill roughly 7×, from about $3,050 a month to around $420, by moving inference to hosted APIs. The write-up is one person's home lab, so treat the exact dollar figures as anecdote. But the conclusion isn't an outlier. Every honest total-cost analysis published over the last year lands in the same place, and it's worth being precise about *why*, because the reason changes how you should decide. ## You're not buying compute, you're buying utilization The instinct behind self-hosting is sound-sounding arithmetic: an A100 80GB rents for roughly $1.10–$2.00/hour on-demand — as low as $0.40–$0.90 on marketplaces like [Vast.ai](https://vast.ai) — and an H100 runs anywhere from about $1.50 to $7 depending on provider. Divide your monthly token volume by what that card can serve and the per-token cost looks great. That arithmetic assumes the GPU is busy. Real traffic isn't. Chat and agent workloads are bursty, diurnal, and spiky around deploys; the dev.to author measured 60% utilization on a deliberately tuned setup, and most teams do far worse. At 10% utilization, every per-token number you calculated is 10× worse. Meanwhile the API provider on the other side of the comparison is running the *same open weights* on the same class of hardware — but batched across hundreds of tenants with continuous batching in [vLLM](https://docs.vllm.ai) or SGLang, at utilization you will never reach with one customer's traffic. They're selling you their batch density, and they can sell it below your marginal cost while still making money. Then there's the price war on top. [DeepSeek](https://api-docs.deepseek.com) has repeatedly cut prices and discounts off-peak traffic; its current V4 Flash tier prices output tokens in the tens of cents per million, and small open models — Qwen3-8B class, GLM-4 class — rent for pennies per million on aggregators like [OpenRouter](https://openrouter.ai). Independent TCO analyses that model this properly find self-hosting doesn't reach parity with hosted open-weight APIs until somewhere around 15–50M sustained tokens per day. Compare against frontier closed-model pricing instead and self-hosting "wins" at a few million tokens a day — but that's the classic apples-to-oranges error. The honest comparison is against the cheapest credible host of the exact model you'd run yourself. ## The GPU is barely half the bill The second thing the home-lab experiment got right — and the part most napkin math omits entirely — is that the GPU line item was only about 55% of real spend. Load balancing, monitoring, on-call time, and model churn made up the rest. Independent analyses are even harsher, putting raw GPU cost at 30–40% of true self-hosting spend once engineering time is priced in. Model churn deserves special attention because it's new. A strong open model now drops every few weeks — Qwen3 variants, GLM updates, DeepSeek point releases, ByteDance's Seed-OSS line. Each has a different VRAM footprint, quantization sweet spot, and serving config. On an API, upgrading is changing a string. On your own cluster, it's a re-provisioning project, and the pace isn't slowing down. Self-hosting doesn't just cost you this month's ops time; it taxes every future model swap. ## Where self-hosting still genuinely wins None of this means the GPU market is only for API providers. Four cases survive the math: **Data residency and compliance.** If prompts legally cannot leave your infrastructure, the economics are irrelevant. This is the single most common legitimate reason, and it's a constraint, not a cost calculation.**Saturated batch workloads.** Embeddings, synthetic-data generation, bulk classification — anything you can queue and run at near-100% utilization on spot-priced marketplace GPUs inverts the whole argument. Utilization was the problem; batch jobs solve it by construction.**Custom weights.** Once you've fine-tuned, hosted per-token pricing for bespoke models gets much worse, and serving your own checkpoint on your own card starts looking sane.**Hedging.** Today's open-model token prices are partly loss-leader pricing in a share-grab. If your product's unit economics depend on $0.66/M output tokens, you want a rehearsed path back to self-hosting before your provider repriced. Notice what's not on the list: "it's cheaper." At interactive-traffic shapes, for almost everyone, it isn't. ## How to actually run your numbers If you're making this call, three concrete moves: First, measure before modeling. Pull thirty days of real token counts from your gateway logs and look at the *shape*, not just the total — peak-to-trough ratio is the utilization number in disguise. A workload averaging 20M tokens/day with 8× daily peaks needs capacity provisioned for the peak, which torches the self-hosting math. Second, compare against the right baseline: the cheapest reliable host of the specific open model you'd deploy, not GPT-5-tier pricing. That single choice moves the break-even by an order of magnitude. Third, keep the switch cheap. Serve everything through an OpenAI-compatible interface — vLLM speaks it natively, and every serious aggregator does too — so "self-hosted vs. API" stays a base-URL change instead of an architecture decision. That's also your hedge from the list above, and it costs almost nothing to maintain. The uncomfortable summary for anyone who enjoys running infrastructure (guilty): open weights being free was never the same thing as open inference being cheap. The weights were commoditized; the utilization wasn't. Until your traffic is big enough or batchable enough to keep a card genuinely busy, the API providers are running your GPU better than you will — and charging you less than it costs you to run it badly. ## Sources & further reading - [I Ran the Numbers on Open-Source AI APIs vs GPUs — Here's the Verdict](https://dev.to/eagerspark/i-ran-the-numbers-on-open-source-ai-apis-vs-gpus-heres-the-verdict-58dp)— dev.to - [Local LLMs vs Cloud APIs: 2026 Total Cost of Ownership Analysis](https://www.sitepoint.com/local-llms-vs-cloud-api-cost-analysis-2026/)— sitepoint.com - [Self-Hosting an LLM vs. API: Real Cost Math (2026)](https://cloudzy.com/blog/self-hosting-open-weight-llm-gpu-vps-cost/)— cloudzy.com - [H100 Rental Prices Compared Across 15+ Cloud Providers (2026)](https://intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison)— intuitionlabs.ai - [DeepSeek API Models and Pricing](https://api-docs.deepseek.com/quick_start/pricing)— api-docs.deepseek.com [Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop. ## Discussion 0 No comments yet Be the first to weigh in.