Tokens per Second Is a Memory Bandwidth Number
Tokens per second for local LLM generation is primarily a memory bandwidth number, not a measure of silicon cleverness, according to an analysis by an unnamed author. At batch size one, generation speed equals memory ban…