Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware

A preliminary benchmark from arXiv (2608.00008) found that Gemma3:1B and Llama3.2:1B models achieve over 170 tokens per second on a single RTX 4060Ti GPU while consuming only 0.56–0.65 joules per token, making them 4.4 times more energy-efficient than the 7B-Mistral model. The findings suggest that smaller, quantized models can significantly reduce energy costs for local AI deployments, though larger models require careful cost-performance tradeoffs.

arXiv https://arxiv.org/abs/2608.00008 Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated. Gemma-1B and LLaMA-1B models can achieve 170 tokens/sec on a single RTX 4060Ti while consuming as little as 0.56-0.65 joules per token, making them viable for high-throughput local deployments where energy efficiency matters. The 7B-Mistral model draws 4.4x more power per token, which means scaling to heavier workloads will require careful cost/performance tradeoff analysis—especially when deploying multiple concurrent agents. Gemma3:1B and Llama3.2:1B hit 0.56–0.65 J/token on a single RTX 4060Ti—4.4× more efficient than 7B-Mistral—while pushing 170 tok/s. This means you can slash cloud GPU spend or run 4× more local inference on the same power budget, but only if you swap out larger models for these smaller, quantized architectures.