{"slug": "mitigating-position-bias-in-transformers-via-layer-specific-positional-embedding", "title": "Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling", "summary": "Researchers introduced LPES, a layer-specific positional embedding scaling method that mitigates the 'lost-in-the-middle' problem in LLMs by assigning distinct scaling factors to each layer, achieving up to 11.2% accuracy gain on key-value retrieval without fine-tuning or latency increase.", "body_md": "arXiv:2606.27705v1 Announce Type: new\nAbstract: Large Language Models (LLMs) still struggle with the ``lost-in-the-middle'' problem, where critical information located in the middle of long-context inputs is often underrepresented or lost. While existing methods attempt to address this by combining multi-scale rotary position embeddings (RoPE), they typically suffer from high latency or rely on suboptimal hand-crafted scaling strategies. To overcome these limitations, we introduce a layer-specific positional embedding scaling~(LPES) method that assigns distinct scaling factors to each layer. LPES achieves a more balanced attention distribution without fine-tuning model parameters or increasing inference delay. A specially designed genetic algorithm is employed to efficiently select the optimal scaling factors for each layer by incorporating B\\'{e}zier curves to significantly reduce the search space. Extensive experiments demonstrate that LPES effectively mitigates positional attention bias and delivers consistent improvements across multiple long-context benchmarks, yielding up to an $11.2$\\% accuracy gain on the key-value retrieval dataset.", "url": "https://wpnews.pro/news/mitigating-position-bias-in-transformers-via-layer-specific-positional-embedding", "canonical_source": "https://arxiv.org/abs/2606.27705", "published_at": "2026-06-29 04:00:00+00:00", "updated_at": "2026-06-29 04:08:00.431138+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "machine-learning", "ai-research"], "entities": ["LPES", "RoPE"], "alternates": {"html": "https://wpnews.pro/news/mitigating-position-bias-in-transformers-via-layer-specific-positional-embedding", "markdown": "https://wpnews.pro/news/mitigating-position-bias-in-transformers-via-layer-specific-positional-embedding.md", "text": "https://wpnews.pro/news/mitigating-position-bias-in-transformers-via-layer-specific-positional-embedding.txt", "jsonld": "https://wpnews.pro/news/mitigating-position-bias-in-transformers-via-layer-specific-positional-embedding.jsonld"}}