Why sinusoidal activations matter #
The core idea behind KANs is replacing the fixed activation functions on nodes with learnable functions on the edges. In the standard implementation, these are usually B-splines. However, B-splines can be computationally heavy and sometimes struggle with extrapolation outside their defined grid. SineKAN replaces these with sine functions, which effectively turns the network into a series of learnable frequency modulations.
From a technical standpoint, this is an interesting pivot. Sinusoids are naturally suited for capturing periodic patterns, but they also have a unique way of representing high-frequency components that splines might need many more parameters to capture. If you are building an AI workflow for physics-informed neural networks (PINNs) or signal processing, this approach is significantly more intuitive than forcing a spline to fit a wave.
Practical implementation and deployment #
If you want to get a hands-on guide on how this differs from a standard MLP or KAN, the main shift is in the weight update. Instead of updating a spline coefficient, the network optimizes the frequency and phase of the sine wave. This can lead to much faster convergence on specific types of mathematical functions.
For those looking for a deep dive into the architecture, the implementation usually follows this logic:
-
The input is multiplied by a learnable weight (frequency).
-
This value is passed through a $\sin(x)$ function.
-
A residual connection or a linear term is often added to maintain stability and prevent the network from getting stuck in local minima caused by the periodicity.
Comparing SineKAN to Standard KANs #
Computational Overhead: SineKAN is generally leaner because calculating a sine function is computationally cheaper than evaluating a B-spline basis.Parameter Efficiency: It often requires fewer parameters to represent oscillatory functions compared to the grid-based approach of original KANs.Convergence Speed: It can converge faster on periodic datasets but may struggle with monotonic functions that a spline would handle effortlessly.Extrapolation: Sinusoids extrapolate periodically, which is either a huge advantage or a huge liability depending on your specific real-world data.
For anyone trying to implement this from scratch, the GitHub repository provides a solid starting point for deployment. The math is handled in the activation layer, meaning you can swap it into existing KAN frameworks relatively easily. It's a great example of how small changes in the activation function can fundamentally alter the behavior of an LLM agent or a regression model.
https://arxiv.org/abs/2407.04149
Stop letting your GPU idle while your CPU struggles to feed it 22h ago
Stop expecting LLMs to be databases because they are 2d ago
Since the provided content was only a title 3d ago
DeepMind WeatherNext actually predicts cyclones with scary 9d ago
Meta's AI actually broke into another company's system during a 10d ago
Meta AI accidentally hacked another company and it's a wild look 11d ago
Next Fitting a diffusion model into 264KB of RAM is actually possible →