Mac Studio Ultra

Tokens per second for local LLM generation is primarily a memory bandwidth number, not a measure of silicon cleverness, according to an analysis by an unnamed author. At batch size one, generation spe…

Local LLM inference splits into two phases—prompt processing (compute-bound) and generation (memory-bandwidth-bound)—explaining why hardware with identical token generation speeds can have vastly diff…