Kalman Delta Networks: Uncertainty-aware Associative Memory

Researchers introduced Kalman Delta Networks (KDN), a linear-attention architecture that models associative memory as an online Bayesian filtering problem, using a Kalman filter to update memory with uncertainty-aware gating. The method, detailed in a paper, targets frontier language models to improve long-context inference and constant-memory decoding by addressing overwrite decisions before full context is known.

Linear attention is increasingly used in frontier language models for efficient long-context inference and constant-memory decoding. Its fixed-size recurrent memory, however, requires an online decision at each token: what to write and how strongly to overwrite existing associations before knowing w