{"slug": "efficient-streaming-language-models-with-attention-sinks", "title": "Efficient Streaming Language Models with Attention Sinks", "summary": "A 2024 paper on attention sinks found that evicting the first few tokens from a language model's KV cache causes performance to collapse abruptly, and proposed a Rolling KV Cache that retains the first 4 tokens to preserve the effect. In a replication with GPT-2 and a sliding-window attention mechanism, next-token perplexity jumped from about 10 to over 3000, with the breakdown beginning at token 257 even though the model's context window is 1024 — showing the cause is loss of the initial tokens, not exceeding the trained context length. The authors found the effect held even when the leading tokens were only newline characters, evidence that models use early tokens as a catchall for softmax attention weight.", "body_md": "Large Language Models (LLMs) are inevitably applied to tasks that take them past the context window they were trained on. This is apparent in long-running chat sessions or parsing large documents and code bases, however, initially the model would drift and performance would significantly deteriorate.\n\nOne of the biggest improvements came from this 2024 paper that popularised the idea of tokens acting as attention sinks. The idea is fairly simple, the model as it causally decodes the sequence starts with a very short set of tokens, and they therefore naturally absorb the entire softmax weight. Naively we’d expect that as the sequence got longer this initial cluster would be spread over the entire sequence and would give us the pattern we expect from textbook explanations, namely that each token has a weighted map over the other tokens in the sequence that are important to it. The authors found this was not the case.\n\nThrough investigation of the KV cache, effectively the context window for the model that can fit into memory, they found that once the initial tokens in the cache started to get evicted the performance degraded significantly and abruptly. This is a weird effect. We could happily imagine if the context window lost a core piece of information essential to decoding the sequence that we’d see the effect, but the authors saw this at the scale of single digit tokens being lost. Additionally they found this to be true even if the tokens contained at the start of a passage of text were simply newline tokens, not anything that could change the meaning of the information of the text. This therefore was evidence of a process in how the LLM was decoding.\n\nThe hypothesis is that the LLM uses these first tokens as something of a catchall for all of the attention weight, because of the softmax, the attention still needs to sum to one. There are plenty of tokens that don’t really matter, and while they could have their weight spread evenly through the sentence, this is a relatively brittle thing to do and varies at each sequence length. So alternatively they could also suppress the weight against most tokens and put all the distribution into what looks like a dummy token at the start of the sequence. These sinks form at the start of the sentence simply because these tokens are visible to all future tokens thanks to the autoregressive nature of the training.\n\nWe can replicate the key result in the paper using a very simple GPT-2 model and a sliding window attention mechanism. We take the model and decode the sequence, for the vanilla case this just carries on as normal with no substantial variation. If instead we enforce a sliding window and start evicting from the KV-cache as we reach a limit, we see the perplexity of the next token explode. The perplexity climbs from an expected ~10 to over 3000. The most important result here though is not the scale of the breakdown but exactly where it happens, even though the context window of the model is 1024 we see this collapse from token 257 because that’s the size of the KV cache we define. This shows that the cause of the collapse is not the sequence getting past the length it was trained on, but is caused by the loss of these first tokens in the sequence.\n\nThe way the authors proposed this work should be interpreted was twofold. First, that any previously trained model could utilise the inbuilt attention sink pattern to improve performance with a small modification they called a Rolling KV Cache. Unlike the typical approach where the cache fills up to some maximum capacity then starts operating like a double-ended queue, popping the earlier elements out of the cache as new tokens are added. They say that the cache maintains the first 4 tokens which cover the vast majority of the attention sink effect. The important extra detail here is that the virtual position of these tokens needs to be updated as the cache moves through the sequence such that the sink tokens remain at positions (in their example) 0 - 3, and then the following window carries on from 4. This is a slightly subtle feature, and does indicate a certain brittleness in the approach. It’s also counter to how the relative position normally works as the window slides through the text. The reason is intuitive though, the sinks operate in a domain where the absolute distance between the sink tokens and the decoding tokens is the cache size, by construction. So it’s not the importance of these tokens per se, but their importance at the start of the known window. Still, it’s a relatively modest adaptation to the inference of a model and can be applied post-hoc to any LLM.\n\nThe second way the work was intended to be taken was to start building these sink tokens into the model training rather than just finding them after the fact. While the use of a Softmax-off-by-one function was proposed to circumvent the problem of requiring the softmax probability to sum to 1, this didn’t really catch on. The alternative of adding explicit sink tokens to the start of the sequence and rolling these into the sliding KV-cache window did. Many models trained in the last couple of years have used this extensively.\n\nOne number that often gets repeated from this paper, incorrectly, is the 4M token context window. It's not actually a coherent token context window, but rather the length of the stream over which the model can maintain stable language modelling performance.\n\nOnce the evicted tokens are gone the model doesn't know anything about them, so any kind of retrieval or dependence on those tokens will break down. What was recovered was the ability to maintain coherent language modelling over very long streams, even if the model has no memory of the earlier parts of the text, once you save the initial sink tokens.", "url": "https://wpnews.pro/news/efficient-streaming-language-models-with-attention-sinks", "canonical_source": "https://idlemachines.co.uk/essays/streaming-llm-attention-sinks", "published_at": "2026-10-30 09:09:58+00:00", "updated_at": "2026-10-30 09:46:56.461207+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "natural-language-processing"], "entities": ["GPT-2", "Rolling KV Cache", "KV cache", "attention sinks"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/efficient-streaming-language-models-with-attention-sinks", "markdown": "https://wpnews.pro/news/efficient-streaming-language-models-with-attention-sinks.md", "text": "https://wpnews.pro/news/efficient-streaming-language-models-with-attention-sinks.txt", "jsonld": "https://wpnews.pro/news/efficient-streaming-language-models-with-attention-sinks.jsonld"}}