{"slug": "how-profitable-is-llm-inference-doing-the-math-on-kimi-k3", "title": "How Profitable is LLM Inference? Doing the Math on Kimi K3", "summary": "LLM inference profitability depends on the trade-off between batch size and GPU count, which determines token latency and cost per million tokens. Applying this model to Kimi K3, which requires at least 16 B200 or 8 B300 GPUs and costs $3 per million input tokens, $15 per million output tokens, and $0.30 per million cached tokens on OpenRouter, the Pareto frontier of speed versus cost defines the optimal pricing and profitability for the service.", "body_md": "# How Profitable is LLM Inference? Doing the Math on Kimi K3\n\nA look at LLM inference economics (batch size, GPU count, and the Pareto frontier that sets token prices) applied to Kimi K3 with back-of-the-envelope math.\n\n## Intro\n\nHow profitable is LLM inference?\n\nThis is a central question sitting at the heart of the AI mania.\n\nIn this post, we’ll cover token pricing basics, build up some intuition, then apply it to the recently released Kimi K3 to answer the profitability question.\n\n## LLM Pricing TLDR\n\nTo produce tokens, we need GPUs and model weights. We load the weights onto one or many GPUs. Large frontier models won’t fit on a single GPU: Kimi K3 needs 16 B200s or 8 B300s at minimum.\n\nGPUs can be rented by the hour. Right now we can head to Lambda or Runpod and rent a B300 for ~$7/hour (at least 8 of those to run Kimi).\n\nTokens are billed by the million. Output tokens are what the model predicts, input tokens are the prompt. Both need processing: one forward pass can process all input tokens at once, while each output token needs its own pass (spec decoding changes this, though). Once input tokens are processed, they generate an internal state (KV cache) that gets reused, so it’s never recomputed. Cached input tokens are cheapest: no GPU compute, just a load from cache (GPU VRAM, RAM, or even colder storage like SSD).\n\nHere’s Kimi K3 pricing on OpenRouter:\n\n1M input tokens cost $3, output is $15, cached tokens are $0.30.\n\nWhere do these prices come from? Here’s a simple model.\n\nGPUs produce tokens. At the margin, some amount of GPU time produces some amount of tokens:\n\n```\n1\ncost per M tokens = hourly_GPU_price / tokens_per_gpu_hour * 1,000,000\n```\n\nThat’s the whole business. Now the central question is: **what determines tokens_per_gpu_hour?**\n\n## The b and N_GPU Situationship and Pareto Frontier\n\nThere are two crucial knobs:\n\n**batch size (b):** how many requests you process together through the same weights in the same forward pass**number of GPUs (N_GPU):** how many chips we spread one model across\n\n```\n1\ntoken_latency = fn(b, N_GPU)\n```\n\n`token_latency`\n\nis a function of the batch size and number of GPUs. `token_latency`\n\nis how long, in seconds, one forward pass takes, the time to produce one new token for every sequence in the batch, i.e. `b`\n\ntokens.\n\nWe can then determine how many tokens a fleet of GPUs cranks out per hour:\n\n```\n1\ntokens_per_gpu_hour = (b / token_latency) / N_GPU * 3600\n```\n\nWe divide by `N_GPU`\n\nbecause that latency was bought with `N_GPU`\n\nchips at once, so each chip is only accountable for its share of the output.\n\nOne more crucial thing:\n\n`token_latency`\n\nalso determines speed. `1/token_latency`\n\nis how many tokens/second each user experiences: lower latency, faster things are. Usually, lowering `b`\n\n(batch size) at a fixed number of GPUs increases speed. So we trade throughput (total tokens) for speed. And fewer tokens per unit time means more expensive tokens.\n\nEach `(b, N_GPU)`\n\npair maps to a `(token_latency, cost)`\n\npair, since together they set both speed/throughput and price.\n\n`b`\n\nand `N_GPU`\n\nare in what you could call a situationship:\n\n- Bigger\n`N_GPU`\n\nmeans more aggregate compute and memory bandwidth -> bigger batch. - Bigger\n`N_GPU`\n\nalso means more aggregate memory, so more room for KV cache -> bigger batch. - At the same time, bigger\n`N_GPU`\n\nmeans more network overhead, since coordination has to travel across all the GPUs -> higher latency. - And bigger\n`b`\n\nmeans more data that has to move across the network -> higher latency.\n\nThe Pareto frontier is the set of `(b, N_GPU)`\n\nand their corresponding`(token_latency, cost)`\n\npairs where you can’t move one without making the other worse: slower or more expensive.\n\nThe cost/speed frontier is probably the key graph for determining profitability. Luckily, the[ vLLM announcement blog ](https://vllm.ai/blog/2026-07-27-k3#serving-performance)has one:\n\nEach point on the frontier gives the optimal price for that speed. Looking at the OpenRouter pricing above, Fireworks offers two versions of Kimi: regular and fast, with fast costing $22 per million tokens instead of $15 but at higher tok/s (lower `token_latency`\n\n). They’re likely running a smaller batch, or more GPUs for the same batch size, buying speed at a higher cost.\n\n## How profitable is Kimi K3 inference\n\nKimi K3 is a ~2.8 trillion parameter model. Quantized to MXFP4, the weights come out to about 1.4TB: too big for one GPU. A B300 has 288GB of memory, 8 of them gives 2.3TB, enough for the weights plus KV cache headroom.\n\nAt the Pareto-optimal point on the cost/speed frontier, this setup gets roughly 30 tok/s for a single user, and about 1,500 tok/s of aggregate throughput per GPU (that’s batching at work — no single user sees 1,500 tok/s, but each GPU serves that many tokens/sec across everyone combined).\n\n```\n1\ntokens_per_gpu_hour = 1,500 * 3,600 = 5,400,000 tokens/hour/GPU\n```\n\nB300s rent for about $7.39/hour:\n\n```\n1\ncost per M tokens = $7.39 / 5,400,000 * 1,000,000 ≈ $1.37 per million tokens\n```\n\n## Too good to be true\n\n$1.37 per million tokens is very far from the advertised $15. Are inference providers laughing their way to the bank? Maybe a chuckle, not a laugh. Two big things dampen the joy:\n\n**Utilization won’t be 100%.** You provision for peak demand, not average. Demand is bursty: coding models see less activity nights, weekends, holidays. The larger the provider, the more demand smooths out across many peaks. But assume 50% utilization and cost doubles immediately.**This is only marginal cost** the cost of producing extra tokens. There’s also salaries, other IT costs, and most importantly, training and fine-tuning. A model is a depreciating asset. Open weights are everywhere right now, but someone shoulders the training cost, and that has to be accounted for somewhere.\n\nThe vLLM post mentions speculative decoding with DSpark giving a ~3x speedup which is tremendous and that’s just one optimization vector. KV cache offloading, prefill/decode disaggregation, and others are actively being worked on. Any inference provider might be sitting on a secret edge that makes the economics far more lucrative than this back-of-the-envelope math suggests.\n\n[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)by the author.", "url": "https://wpnews.pro/news/how-profitable-is-llm-inference-doing-the-math-on-kimi-k3", "canonical_source": "https://cefboud.com/posts/ai-inference-costs-profit/", "published_at": "2026-07-29 00:00:00+00:00", "updated_at": "2026-07-29 05:29:27.502993+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-products"], "entities": ["Kimi K3", "OpenRouter", "Lambda", "Runpod", "B200", "B300", "vLLM"], "alternates": {"html": "https://wpnews.pro/news/how-profitable-is-llm-inference-doing-the-math-on-kimi-k3", "markdown": "https://wpnews.pro/news/how-profitable-is-llm-inference-doing-the-math-on-kimi-k3.md", "text": "https://wpnews.pro/news/how-profitable-is-llm-inference-doing-the-math-on-kimi-k3.txt", "jsonld": "https://wpnews.pro/news/how-profitable-is-llm-inference-doing-the-math-on-kimi-k3.jsonld"}}