Speeding Up JumpReLU SAE Inference with Custom Triton Kernels (2–14× on Real SAEs)
Researchers developed custom Triton kernels that accelerate JumpReLU Sparse Autoencoder inference by 2–14× on real SAEs, exploiting activation sparsity to skip zero entries during matrix multiplication. The kernels use e…