FourierQK: Filter Shape, Admissibility and the Leakage-Coverage Law
A controlled ablation on character-level language modelling with a 6-layer GPT on TinyShakespeare found that the optimal single-scale bandwidth for FourierQK bandpass-filtered attention is sigma ~= 2 …