{"slug": "linear-time-context-compression-with-state-space-encoding", "title": "Linear-Time Context Compression with State-Space Encoding", "summary": "A new arXiv paper (2610.33006v1) proposes the Boa Compressor, a plug-and-play online context compressor built on a Mamba state-space model that encodes arbitrary-length context in a single linear-time pass and conditions a frozen target LLM on the resulting representation. Trained by distillation to match the next-token logit distribution of the full-context teacher, Boa matches or surpasses state-of-the-art offline compression baselines at equal compression ratios while running an order of magnitude faster than Transformer-based implementations, scaling to compression ratios as aggressive as 16x. The authors argue this makes compression cheap enough to run on every query at inference time, reducing end-to-end latency for long-context inference rather than adding compression overhead.", "body_md": "arXiv:2610.33006v1 Announce Type: new \nAbstract: Transformer-based large language models (LLMs) impose substantial computation and memory costs as context sizes increase, motivating methods that compress a context into a shorter representation before inference. Existing soft-prompt compression methods typically use the same Transformer backbone as used in the target model, where producing the compressed representation leads to significant latency overheads. This restricts them to the offline setting where context is compressed once and then reused across many queries. We propose the Boa Compressor, a plug-and-play online context compressor built on a Mamba state-space model (SSM) that encodes arbitrary-length context in a single linear-time pass, producing a representation on which we condition a frozen target LLM. The SSM is trained by distillation, matching the next-token logit distribution from the LLM ingesting the full context (as teacher) to the same LLM ingesting the compressed representation (as the student). At equal compression ratios, Boa matches or surpasses the performance of existing state-of-the-art offline compression baselines, while being an order of magnitude faster than Transformer-based implementations, thereby reducing end-to-end latency rather than adding compression time overhead. We demonstrate that Boa also scales well to longer contexts and to compression ratios as aggressive as 16x. By enabling effective compression that is cheap enough to run on every query at inference time, Boa becomes a practical and general tool for efficient long-context inference.", "url": "https://wpnews.pro/news/linear-time-context-compression-with-state-space-encoding", "canonical_source": "https://arxiv.org/abs/2610.33006", "published_at": "2026-10-27 04:00:00+00:00", "updated_at": "2026-10-27 04:11:55.725237+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "natural-language-processing"], "entities": ["Boa Compressor", "Mamba", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/linear-time-context-compression-with-state-space-encoding", "markdown": "https://wpnews.pro/news/linear-time-context-compression-with-state-space-encoding.md", "text": "https://wpnews.pro/news/linear-time-context-compression-with-state-space-encoding.txt", "jsonld": "https://wpnews.pro/news/linear-time-context-compression-with-state-space-encoding.jsonld"}}