“Running Local Models Is Good Now” Was Written on a 64GB Mac. Half of You Have 16GB or Less

A new article argues that running large language models locally on consumer hardware is now feasible, but notes that 52% of PCs have 16GB RAM or less, limiting model size. The author explains that Mac and PC 16GB RAM are not equal due to differences in memory architecture and KV cache costs.

52% of PCs have 16GB RAM or less. Here’s what local LLMs actually fit, what the KV cache costs you, and why Mac and PC 16GB are not equal. Continue reading on Towards AI »