17:59
2026-09-07
research.colfax-intl.com
machine-learning
Optimization diaries: S/P ping-pong for FlashAttention-4 decode
A new optimization for FlashAttention-4 (FA4) decoding on NVIDIA Blackwell GPUs overlaps softmax and matrix multiplication operations by using spare tensor memory (TMEM) in a ping-pong scheme, achievi…