Kimi K3 Attention Residuals

Kimi K3, an AI model from Moonshot AI, introduces Attention Residuals, replacing standard uniform residual connections in deep transformers with attention-weighted layer outputs, allowing the model to dynamically weight contributions from previous layers. The mechanism, detailed in a report, uses a learned pseudo-query per layer and softmax over dot products to compute weights, with a Block Attention Residuals variant to reduce memory and communication overhead. This architectural change aims to improve gradient flow in 100+ layer models, enhance interpretability via attention weights, and provide richer distillation targets.

Kimi K3 Attention Residuals Standard residuals are dead simple: each layer output gets added to the accumulated sum with equal weight. Expand the recurrence and you get h l = h 1 + Σ f i h i from i=1 to l-1. Every layer contributes exactly once, no questions asked. Works fine, but it's rigid — early layers can't be downweighted if they're noisy, later layers can't be amplified if they carry the real signal. Kimi K3 replaces that fixed sum with attention over layer outputs. The mechanism learns a pseudo-query w l per layer input-independent, so it's really a learned positional bias and computes weights via softmax over dot products with previous layer values v i . The layer output becomes a weighted combination: h l = Σ α i v i where α = softmax w l · v i / √d . Key difference: the model decides how much each predecessor matters. Early token embeddings can be suppressed if they're irrelevant to the current reasoning step. Critical intermediate representations get boosted. It's essentially a learned highway network where the gates are computed via attention rather than a separate MLP. The report notes Kimi K3 actually uses Block Attention Residuals — grouping layers into blocks to cut memory and communication overhead. Full Attention Residuals are the conceptual baseline. Still, the principle holds: instead of h l = h {l-1} + f l h {l-1} , you get dynamic routing. Practical implications for anyone building or fine-tuning deep transformers: Gradient flow : Attention-weighted skips should alleviate vanishing gradients in 100+ layer models better than uniform residuals Interpretability : The attention weights α give you a direct read on which layers contribute to a given prediction — rare for internal mechanisms Distillation target : Teacher models with attention residuals provide richer supervision signals than standard ResNet-style teachers Curious how this compares to DeepSeek /en/tags/deepseek/ 's multi-head latent attention or Google's mixture-of-depths. Both touch on dynamic computation allocation but from different angles. Kimi's approach feels more architectural — changing the backbone rather than adding routing overhead. If you're training deep models from scratch, this is worth experimenting with. The pseudo-query design means minimal parameter overhead one vector per layer and the softmax keeps it stable. Would love to see ablation studies on depth scaling curves vs standard residuals. Next Rust-based k9s replacement cuts cluster navigation latency by 60 → /en/threads/6946/