I spent two weeks optimizing 96GB of VRAM for local LLMs. Paid APIs still won.

A developer spent two weeks optimizing a homelab with four RTX 3090s (96GB VRAM) for local LLM inference, achieving improvements like 40% throughput gain and 4x VRAM savings, but ultimately found that paid APIs were more cost-effective for interactive work due to low GPU utilization (6%) caused by sequential dispatch in llama.cpp. The developer concluded local setups are better for privacy, batch jobs, or uncensored experimentation, but not as a general cloud replacement.

I run a homelab with four RTX 3090s — 96 GB of VRAM, 44 CPU cores. For two weeks I tried to make it my daily driver for local LLM inference instead of paying for cloud APIs. I got it working. Then I looked at the numbers and subscribed to a paid API anyway. Here's the uncomfortable part, and the optimizations that still made it worth doing. The setup The 6% problem The wall wasn't compute. GPU utilization sat at 6%. The bottleneck was CPU orchestration — llama.cpp dispatches across multiple GPUs sequentially, so the cards spent 94% of the time idle waiting on each other. Throwing more VRAM at it does nothing for this. What actually moved the needle | Change | Effect | |---|---| | --ubatch-size 512 | +40% throughput | | KV cache quantization Q4 0 | 4× VRAM savings | | Speculative decoding n-gram | 2.5× speedup on repetitive tasks | | YaRN rope scaling | context extended to 1M tokens | Two things surprised me: The conclusion I didn't want At ~11 kWh/day, plus hardware depreciation, against current API pricing, the math doesn't favor local for interactive work. The single biggest improvement to my daily AI workflow was paying for an API. Local still wins for privacy, high-volume batch jobs, or uncensored experimentation — but not as a general cloud replacement. It's an economics problem, not a capability one. I wrote up the full cost breakdown and the exact llama.cpp router configs on aipster.com https://aipster.com/four-gpus-two-weeks-and-the-uncomfortable-truth-about-local-llms/ . If you're weighing a local rig, I also benchmarked GLM 5.2's open weights https://aipster.com/glm-5-2-open-weight-top-four-model-hugging-face/ — it changed my view on what's worth running at home. What's your GPU utilization actually sitting at? Curious if anyone solved the sequential-dispatch problem.