TinyShakespeare

A controlled ablation on character-level language modelling with a 6-layer GPT on TinyShakespeare found that the optimal single-scale bandwidth for FourierQK bandpass-filtered attention is sigma ~= 2 …

Researchers have developed a spectral preprocessing method using FFT that significantly improves attention mechanisms in character-level language models, achieving up to a 79% reduction in validation …