Understanding FlashAttention Pt 1: Personal Notes
A technical handbook on FlashAttention explains that the algorithm achieves wall-clock speedups by reducing data movement between GPU high-bandwidth memory (HBM) and on-chip SRAM rather than by approximating attention, u…