Photo: Jawed Karim . / Wikimedia Commons / CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0/) A new metric called 'intelligence per joule' reveals how model design and hardware advances are reshaping the economics of local AI inference.
Running a large language model locally used to feel like heating your apartment with a space shuttle engine. New research out of Stanford’s Hazy Research group suggests that equation has changed dramatically: the intelligence you can squeeze out of a single joule of energy has improved 18-fold in roughly 16 months.
The finding comes from a paper titled “Measuring Intelligence Efficiency of Local AI” (arXiv:2511.07885), which introduces two new metrics designed to benchmark local AI inference: Intelligence per Joule (IPJ) and Intelligence per Watt (IPW). Think of IPJ as the miles-per-gallon rating for AI models running on your own hardware rather than in a distant data center.
Where the 18x came from #
The 18x improvement in IPJ from mid-2024 to late 2025 didn’t come from a single breakthrough. It was a compound effect. Model architecture improvements contributed roughly 3.1x of the gain. Hardware and accelerator advances delivered a much larger 5.9x boost.
On the model side, the research tracked a progression from earlier architectures like Mixtral-8x7B to newer mixture-of-experts variants and models like gpt-oss-120b. These designs route computation more selectively, activating only the parameters needed for a given query rather than lighting up the entire network every time.
The hardware story is arguably more interesting. The paper benchmarked a range of chips including NVIDIA’s Quadro RTX 6000, Apple’s M4 Max, NVIDIA’s B200, and SambaNova’s SN40L. Purpose-built AI accelerators like the B200 and SN40L outperformed mainstream consumer silicon like Apple’s M4 Max on efficiency metrics.
By the end of the evaluation period, local language models had reached approximately 88.7% accuracy on single-turn chat and reasoning queries.
The hybrid model changes the cost equation #
The paper found that hybrid local-cloud routing, where simpler queries are handled on-device and only complex ones get sent to the cloud, can reduce energy consumption, compute requirements, and costs by 60-80% compared to running everything through cloud-based models, without meaningful sacrifices in answer quality.
To put that in practical terms: if a company is spending $10 million annually on cloud AI inference, a well-implemented hybrid approach could potentially trim that to $2-4 million while delivering comparable results. The IPW metric showed a 5.3x increase for local AI over a two-year window. While IPJ captures total energy efficiency per task, IPW reflects how much useful work a system produces for every watt it continuously consumes.
The profiling harness released alongside the paper went public around November 2025, giving teams a concrete tool to benchmark their own setups against these efficiency frontiers.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our