Attention Sinks: Why Streaming LLMs Break When You Evict Token 0

A developer explains the phenomenon of 'attention sinks' in large language models, where the first few tokens in a sequence receive disproportionate attention weight, even when semantically irrelevant. This causes naive sliding-window KV cache eviction to fail catastrophically, as models like Llama, Mistral, and GPT rely on these sink tokens for stable generation. The root cause is standard softmax attention, which forces attention weights to sum to 1, leading models to dump excess probability mass on early tokens as a no-op.

Drop the first four tokens from a sliding-window KV cache and your model's perplexity doesn't degrade gracefully — it detonates. Generation turns to garbage within a few steps, even though those four tokens were a <bos marker and the word "The." That failure is the fingerprint of an attention sink , and if you serve long-context or streaming LLMs, it dictates exactly which parts of the KV cache you are allowed to throw away. This is the mechanism behind StreamingLLM, and it's the reason naive sliding-window attention silently corrupts output. An attention sink is a token position — almost always among the first few in the sequence — that receives a large, roughly constant fraction of attention weight from most heads in most layers, even when it carries no semantic relevance to the query. In a trained transformer, plot the attention maps and you'll see a bright vertical stripe at position 0: nearly every later token routes a chunk of its attention there. The tokens themselves are unremarkable — a beginning-of-sequence token, a newline, "The." Their content doesn't matter. Their position does. Because attention is causal, the first tokens are the only keys visible to every subsequent query. Over training, the model learns to use them as a shared, always-available anchor. Because softmax attention is forced to allocate exactly 100% of its weight, whether or not any key deserves it. Look at the operation: attn = softmax Q Kᵀ / √d V The denominator normalizes the exponentiated scores so the row sums to 1. When a query has no strong match among the keys — which happens constantly in deeper layers, where a head may simply have nothing to do for this token — every logit is small and similar. Softmax can't output "attend to nothing." It still produces a distribution that sums to 1, so it spreads that mass somewhere. The model's solution is to designate a scapegoat. It pushes the excess onto the earliest tokens and learns to make their value vectors near-zero , so attending to a sink contributes almost nothing to the output. The sink becomes a functional no-op: a place to park probability mass so the real keys aren't polluted by noise. This lines up with the "massive activations" observation — certain hidden dimensions at sink positions carry enormous magnitude relative to everything else. Those outliers are the physical implementation of the sink. They're also why per-tensor activation quantization on these positions is dangerous. Evan Miller framed the root cause well: standard softmax has no escape hatch. The proposed softmax1 adds a +1 to the denominator: softmax1 x i = exp x i / 1 + Σ j exp x j Now the row can sum to less than 1, letting a head attend to almost nothing. Models trained with this variant, or with an explicit learnable sink logit in the denominator, don't need to hijack token 0. But every pretrained model you're serving today — Llama, Mistral, Qwen, GPT-family — was trained with plain softmax and does rely on positional sinks. Because it eventually evicts the sink tokens, and the model was never trained to run without them. This is the failure that StreamingLLM Xiao et al., 2023 diagnosed. The setup is standard for streaming: you want to generate indefinitely without the KV cache growing without bound, so you keep only the most recent W tokens' keys and values. For the first W tokens everything is fine. The moment generation passes W and the window starts sliding, position 0 falls out of the cache. Now the queries that learned to dump ~10-40% of their attention on the sink have nowhere to put it. Softmax renormalizes over the remaining in-window keys, and that parked mass — which was supposed to be a no-op — lands on genuinely attended tokens with the wrong weights. Hidden states shift out of distribution, the next layer amplifies the error, and perplexity spikes hard. It doesn't recover, because every subsequent step inherits the corrupted cache. The counterintuitive part: dense attention over the same W recent tokens recomputed each step, no sliding cache works fine. Sliding-window caching over the same tokens breaks. The only difference is whether positions 0..3 are present. Pin the first few tokens in the cache permanently and slide the rest. StreamingLLM keeps roughly 4 initial sink tokens plus a rolling window of the most recent tokens. That's the whole fix — no fine-tuning, no architecture change. Perplexity stays flat over sequences millions of tokens long, and memory stays bounded. Here's the eviction logic: python class SinkKVCache: def init self, n sink=4, window=2048 : self.n sink = n sink self.window = window self.keys = per-layer tensors, shown flat for clarity self.vals = def append self, k, v : self.keys.append k self.vals.append v cap = self.n sink + self.window if len self.keys cap: keep 0 : n sink + -window : self.keys = self.keys :self.n sink + self.keys -self.window: self.vals = self.vals :self.n sink + self.vals -self.window: One detail that trips people up: positions . When you concatenate sink tokens with a distant window, don't feed the sinks their original absolute positions. With RoPE, apply position IDs based on the cache layout , not the token's place in the original stream — the sinks sit at positions 0..3 and the window continues from there. Encoding the true 2-million-token gap between them re-introduces the out-of-distribution problem you were trying to avoid. If you control training, the cleaner fix is a dedicated learnable sink token prepended to every sequence, or an explicit sink logit added to each head's softmax denominator the softmax1 idea, per-head . Then no real token gets hijacked and eviction of ordinary tokens is safe. Several recent open models bake this in. Sink tokens are load-bearing, so any policy that compresses or drops KV entries has to treat positions 0..3 as untouchable. The general rule: token 0 is not just data, it's structural. Treat the first handful of positions as part of the model's machinery, not as evictable history. Streaming LLMs break when you evict the first tokens because those tokens are attention sinks — the positions every later query dumps its leftover softmax probability onto, since softmax must sum to 1 and a query with nothing relevant to attend to still has to put its mass somewhere. The model learns to make these sinks near-zero-value no-ops. Remove them from a sliding-window KV cache and that parked probability mass gets redistributed onto real tokens with the wrong weights, pushing hidden states out of distribution and spiking perplexity within a few steps. The fix is to pin roughly four initial sink tokens in the cache alongside your rolling window and assign them cache-layout positions, not original ones , which keeps a streaming LLM stable over millions of tokens without retraining. Any KV-cache eviction or quantization policy you deploy must treat those sink positions as non-evictable and outlier-aware, or your model's accuracy will quietly collapse the moment the window starts to slide.