Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings

Researchers proposed SCSP, a training-free framework that uses semantic compression prompts to select informative tokens for long-context embeddings instead of uniformly averaging all token representations, according to an arXiv paper (arXiv:2609.37782v1). SCSP partitions documents into sentence-aware chunks, appends a semantic compression prompt to each chunk, and uses a prompt-isolated attention mask to keep each prompt restricted to its local context while preserving information flow among document tokens. The paper reports that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner and consistently improves performance on long-context embedding benchmarks.

arXiv:2609.37782v1 Announce Type: new Abstract: Large language models LLMs have shown strong potential as training-free text encoders for long-context embeddings. Existing approaches primarily improve information flow under causal attention and typically construct embeddings by uniformly averaging all token representations. However, for long documents, such mean pooling can dilute salient semantic information with abundant redundant or weakly informative content. To this end, we propose SCSP, a training-free framework that leverages semantic compression for informative token selection in long-context embedding. Specifically, SCSP first partitions a document into sentence-aware chunks and appends a semantic compression prompt to each chunk. A prompt-isolated attention mask preserves information flow among document tokens while restricting each prompt to its corresponding local context. We then use the attention patterns elicited by these prompts to estimate token importance, select informative tokens, and aggregate their intermediate-layer representations into the final embedding. Extensive experiments on long-context embedding benchmarks demonstrate that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.