I run a homelab with four RTX 3090s — 96 GB of VRAM, 44 CPU cores. For two weeks I tried to make it my daily driver for local LLM inference instead of paying for cloud APIs. I got it working. Then I looked at the numbers and subscribed to a paid API anyway.
Here's the uncomfortable part, and the optimizations that still made it worth doing.
The setup #
The 6% problem #
The wall wasn't compute. GPU utilization sat at 6%. The bottleneck was CPU orchestration — llama.cpp dispatches across multiple GPUs sequentially, so the cards spent 94% of the time idle waiting on each other. Throwing more VRAM at it does nothing for this.
What actually moved the needle #
| Change | Effect |
|---|---|
| `--ubatch-size 512`
| +40% throughput |
| KV cache quantization (Q4_0) | 4× VRAM savings |
| Speculative decoding (n-gram) | 2.5× speedup on repetitive tasks |
| YaRN rope scaling | context extended to 1M tokens |
Two things surprised me:
The conclusion I didn't want #
At ~11 kWh/day, plus hardware depreciation, against current API pricing, the math doesn't favor local for interactive work. The single biggest improvement to my daily AI workflow was paying for an API. Local still wins for privacy, high-volume batch jobs, or uncensored experimentation — but not as a general cloud replacement. It's an economics problem, not a capability one.
I wrote up the full cost breakdown and the exact llama.cpp router configs
[on aipster.com](https://aipster.com/four-gpus-two-weeks-and-the-uncomfortable-truth-about-local-llms/).
If you're weighing a local rig, I also benchmarked
[GLM 5.2's open weights](https://aipster.com/glm-5-2-open-weight-top-four-model-hugging-face/)
— it changed my view on what's worth running at home.
What's your GPU utilization actually sitting at? Curious if anyone solved the sequential-dispatch problem.