TESLA: Taylor Expansion of Sinusoidal Learnable Activations

Researchers at KAU QuantumAILab propose TESLA, a learnable sinusoidal activation that solves the parity problem for binary vectors of length 32 with 100,000 training samples (0.002% of the input space) and maintains high accuracy under up to 30% label noise. The activation, which combines sine and cosine terms with constrained coefficients for Lipschitz bounds, outperforms SIREN, SNAKE, and Fourier feature baselines on parity and Forrelation tasks and achieves comparable performance on ImageNet-100.

arXiv:2608.11970v1 Announce Type: new Abstract: The parity problem--deciding whether the number of ones in a binary vector is odd or even--remains challenging for standard neural networks due to linear inseparability and the need for global interactions. We propose TESLA, an activation defined as a learnable combination of sine and cosine terms, enabling explicit control over polynomial degree and selective amplification of high-order components. Theoretically, we show that constraining TESLA's coefficients yields Lipschitz/Rademacher complexity bounds and shapes the training dynamics to emphasize higher-frequency structure. Empirically, on parity with input length n = 32, TESLA attains strong generalization with 100K training samples approximately 0.002% of the 2^32 input space and remains robust under heavy corruption, retaining high accuracy with up to 30% label noise. We also compare against periodic and frequency-based baselines SIREN, SNAKE, and Fourier feature embeddings on parity and Forrelation. Beyond synthetic structure, TESLA delivers comparable performance on ImageNet-100, indicating that activation-level degree control transfers to more general vision workloads. Code: https://github.com/KAU-QuantumAILab/TESLA