The SIMPLE Law for Energy Cost of LLM Inference

A new analytical model, the SIMPLE Law, addresses the non-linear energy cost of LLM inference by accounting for how each token entering the KV-cache raises the cost of subsequent tokens, according to the article. The article states that no prior analytical model specified what a token costs and that the most accurate state-of-the-art model was previously the benchmark. The work matters because generative AI is billed per token even though tokens are not equally expensive.

Generative AI is billed per token, yet tokens are not equally expensive. Each token enters the KV-cache and raises the cost of those that follow, so energy grows non-linearly along a sequence whose tokens look identical. No analytical model states what a token costs, and the most accurate SOTA model