# Kimi K3 Attention Residuals > Source: > Published: 2026-08-19 16:45:24+00:00 # Kimi K3 Attention Residuals Standard residuals are dead simple: each layer output gets added to the accumulated sum with equal weight. Expand the recurrence and you get `h_l = h_1 + Σ f_i(h_i)` from i=1 to l-1. Every layer contributes exactly once, no questions asked. Works fine, but it's rigid — early layers can't be downweighted if they're noisy, later layers can't be amplified if they carry the real signal. Kimi K3 replaces that fixed sum with attention over layer outputs. The mechanism learns a pseudo-query `w_l` per layer (input-independent, so it's really a learned positional bias) and computes weights via softmax over dot products with previous layer values `v_i` . The layer output becomes a weighted combination: `h_l = Σ α_i v_i` where `α = softmax(w_l · v_i / √d)` . Key difference: the model *decides* how much each predecessor matters. Early token embeddings can be suppressed if they're irrelevant to the current reasoning step. Critical intermediate representations get boosted. It's essentially a learned highway network where the gates are computed via attention rather than a separate MLP. The report notes Kimi K3 actually uses Block Attention Residuals — grouping layers into blocks to cut memory and communication overhead. Full Attention Residuals are the conceptual baseline. Still, the principle holds: instead of `h_l = h_{l-1} + f_l(h_{l-1})` , you get dynamic routing. Practical implications for anyone building or fine-tuning deep transformers: **Gradient flow**: Attention-weighted skips should alleviate vanishing gradients in 100+ layer models better than uniform residuals** Interpretability**: The attention weights`α` give you a direct read on which layers contribute to a given prediction — rare for internal mechanisms**Distillation target**: Teacher models with attention residuals provide richer supervision signals than standard ResNet-style teachers Curious how this compares to [DeepSeek](/en/tags/deepseek/)'s multi-head latent attention or Google's mixture-of-depths. Both touch on dynamic computation allocation but from different angles. Kimi's approach feels more architectural — changing the backbone rather than adding routing overhead. If you're training deep models from scratch, this is worth experimenting with. The pseudo-query design means minimal parameter overhead (one vector per layer) and the softmax keeps it stable. Would love to see ablation studies on depth scaling curves vs standard residuals. [Next Rust-based k9s replacement cuts cluster navigation latency by 60 →](/en/threads/6946/)