# I changed nothing and my LLM server got 27% more expensive > Source: > Published: 2026-09-26 19:38:35+00:00 I ran the same cost check four times in a row against a server I didn't touch. Same model, same machine, same prompts, same settings. This is what came back, in dollars per million output tokens: | Run | $/M output tokens | 95% CI | Change vs previous run | |---|---|---|---| | 1 | $8.77 | $8.19 to $9.35 | | | 2 | $8.86 | $7.80 to $9.93 | +1.0% | | 3 | $11.26 | $10.31 to $12.21 | **+27.1%** | | 4 | $10.02 | $7.41 to $12.62 | **−11.0%** | Run 3 says the server got 27% more expensive. Run 4 says it then got 11% cheaper. Nothing changed. If I had changed a flag between run 2 and run 3, I would have blamed the flag and rolled it back. If I had changed one between run 3 and run 4, I would have posted an 11% saving in Slack. Both would have been wrong, and both would have looked like data. This post is about why that happens and the rule I now apply before believing any before/after cost number. The tool in the output below is [Throttle](https://github.com/KushagraKanaujia/throttle), an open-source CLI I'm building, but the rule doesn't depend on it. A MacBook running Ollama locally with `llama3.2:3b`. Each check sent 3 blocks of 4 requests at concurrency 2, capped at 64 output tokens, temperature 0, with the same 8 built-in prompts and one warm-up request. All four checks finished within about 70 seconds of each other. ``` throttle check --url http://localhost:11434 --model llama3.2:3b \ --gpu-hourly-rate 1.50 --label same \ --blocks 3 --requests-per-block 4 --concurrency 2 --max-tokens 64 ``` The $1.50/hour GPU rate. A laptop has no GPU invoice, so I picked and ran Throttle on runpod on different GPUa labels it ASSUMED everywhere it appears. Tokens and time are measured. The cost math is simple: ``` $/M output tokens = hourly rate × wall-clock hours ÷ output tokens × 1,000,000 ``` Every block in all four runs produced exactly 256 output tokens (4 requests × 64 tokens). So the only thing that moved the price was time. The three blocks of run 1 took 5.53 s, 5.24 s and 5.40 s. The blocks of run 3 took 7.09 s, 7.02 s and 6.65 s. Same tokens, about 30% more seconds, about 27% more dollars per token. A cost measurement is a timing measurement, and timing on a real machine depends on everything else the machine is doing: other processes, background indexing, thermal state and clock speed, memory pressure, OS scheduling. On a shared cloud node you can add noisy neighbours and other tenants on the same host or network. I didn't instrument the laptop during these runs, so I can't tell you which of those it was. That's the point. Most of the time you won't know either, and a single before/after pair can't tell load from a real change. Each check reports a 95% confidence interval across its blocks. Look at runs 1 and 3: $8.19 to $9.35, then $10.31 to $12.21. The intervals don't overlap. The common rule of thumb ("non-overlapping intervals means a real difference") would call run 3 a real 27% regression. The interval answers a narrower question than it looks like. It's computed from the blocks inside one run, so it describes the wobble over about 20 seconds. It can't see drift between runs, because every block in a run shares the same conditions. If the whole machine is 30% slower for a minute, all three blocks are slower together and the interval stays tight around the wrong number. The fix is to measure run-to-run noise directly: run the unchanged config a few times, and see how much whole checks disagree with each other. Say the relative standard deviation between repeat checks of one config is SD. The difference between two single checks then has a standard deviation of √2 × SD, because each check carries its own noise. With only a handful of repeats you use Student's t instead of 1.96, with degrees of freedom from the repeats. That gives a 95% bound on how far apart two checks of the **same** config can land: ``` noise bound = t(0.975, df) × √2 × SD ``` Here, runs 1 to 3 were three checks of the unchanged config: $8.77, $8.86, $11.26. Their relative SD is 14.6%, with 2 degrees of freedom, where t is 4.303: ``` 4.303 × 1.414 × 14.6% ≈ 89% ``` Throttle prints 89.1% because it uses the unrounded SD. Run 4 came in 11.0% below run 3. That's well inside ±89.1%, and the two intervals overlap as well, so this is the verdict: ``` before: $11.26/M output tokens (95% CI $10.31 to $12.21) after: $10.02/M output tokens (95% CI $7.41 to $12.62) change: -11.0% noise floor: calibrated: bound 89.1% (t x sqrt(2) x 14.6% run-to-run SD, 2 df) Verdict: NO WINNER, the change (-11.0%) is not larger than the run-to-run noise bound (89.1% = t x sqrt(2) x 14.6% run-to-run SD, 2 df); and the 95% confidence intervals overlap, so the difference is within measurement noise ``` What about the +27.1% at run 3? Throttle didn't call it anything. With fewer than three earlier checks of the same config there's no noise estimate at all, so runs 2 and 3 both came back as: ``` Verdict: NOT CALIBRATED — run-to-run noise unknown. Two checks at different times can differ from load alone. Run 'throttle check' at least 3 times without changing anything (within 24 h) to measure your noise, or use 'throttle golden' for a counterbalanced decision. ``` I built it that way because refusing to answer beats a confident wrong answer. A bound of ±89% means these three repeats can't resolve any realistic config change, and that's true. One slow run out of three blew the SD up, and with 2 degrees of freedom t is large. The bound is saying: on this machine, right now, you don't know enough to judge a change. You can tighten it in two ways: For a decision you'll act on, like which config goes to production, don't compare two sequential runs at all. Interleave them (baseline, candidate, baseline, candidate, baseline, candidate) so drift hits both sides equally. I wrote up a real six-run example on an A100 in [One vLLM flag, −68.6% cost per token](https://www.throttle-pro.com/blog/one-vllm-flag-counterbalanced-benchmark). This was a laptop, a 3B model, a short workload and an assumed GPU rate. The size of the swing is specific to that machine on that morning. Your servers will have their own noise, which is exactly why you have to measure it rather than borrow mine. These runs used Throttle 0.4.0, which repeats identical prompts, so the prompt cache was warm in all four runs. That rules out the cache as the cause of the swing, since it was the same in every run. Since 0.4.1, Throttle tags every request so a prefix cache can't make later checks look cheaper, and `--warm-cache` measures a warm cache on purpose. ``` pipx install throttle-pro throttle demo # no GPU needed; every number is SIMULATED ``` Then point `throttle check` at your own vLLM, SGLang or Ollama endpoint and run it three times before you change anything. If you get an interesting result, `throttle check --share` prints a sanitized summary you can post as a [results issue](https://github.com/KushagraKanaujia/throttle/issues/new?template=share-results.yml). I'm collecting what real serving changes actually do to cost, and I read every one.