Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
Gemma-1B and LLaMA-1B models can achieve 170 tokens/sec on a single RTX 4060Ti while consuming as little as 0.56-0.65 joules per token, making them viable for high-throughput local deployments where energy efficiency matters. The 7B-Mistral model draws 4.4x more power per token, which means scaling to heavier workloads will require careful cost/performance tradeoff analysis—especially when deploying multiple concurrent agents.
Gemma3:1B and Llama3.2:1B hit 0.56–0.65 J/token on a single RTX 4060Ti—4.4× more efficient than 7B-Mistral—while pushing >170 tok/s. This means you can slash cloud GPU spend or run 4× more local inference on the same power budget, but only if you swap out larger models for these smaller, quantized architectures.