{"slug": "copy-paste-language-models", "title": "Copy-Paste Language Models", "summary": "Researchers propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head alongside the conventional language modeling head, according to arXiv paper 2610.19247v1. Pretraining models from 135M to 1.7B parameters showed improvements in training efficiency and downstream performance across all scales and backbones with negligible training and inference overheads. The authors attribute the gains to CPLMs partially mitigating the LM head gradient bottleneck through an alternative gradient path.", "body_md": "arXiv:2610.19247v1 Announce Type: new \nAbstract: Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.", "url": "https://wpnews.pro/news/copy-paste-language-models", "canonical_source": "https://www.machinebrief.com/news/copy-paste-language-models-h9kc", "published_at": "2026-10-15 04:00:00+00:00", "updated_at": "2026-10-15 05:52:45.503538+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence", "ai-research", "natural-language-processing"], "entities": ["Copy-Paste Language Models", "CPLM", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/copy-paste-language-models", "markdown": "https://wpnews.pro/news/copy-paste-language-models.md", "text": "https://wpnews.pro/news/copy-paste-language-models.txt", "jsonld": "https://wpnews.pro/news/copy-paste-language-models.jsonld"}}