Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens

A proposed method prepends trainable [DUMP] tokens to input text to reduce representational interference in retrieval embedding models, addressing how fixed-size vector compression can entangle semantic content with residual token-level information such as artifacts from attention sink behavior. The approach targets the separation of semantic meaning from token-level noise in embeddings used for retrieval.

Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable DUMP tokens to