{"slug": "does-a-data2vec-auxiliary-help-small-masked-language-models-a-babylm-2026-strict", "title": "Does a Data2vec Auxiliary Help Small Masked Language Models? A BabyLM-2026 Strict-Small Study", "summary": "A data2vec-style EMA self-distillation auxiliary (λ=0.1, τ=0.999) produced no significant improvement in a 33M-parameter LTG-BERT masked language model at the BabyLM-2026 Strict-Small scale, according to a study by Martin Benedikt, Ivan Dostál and Robin Klubarski published in the Proceedings of the Second BabyLM Workshop (pages 631–638). Across 6 seeds, no evaluation task changed significantly (aggregate effect p=0.29), and an early Entity-Tracking gain that motivated the study disappeared on re-evaluation and is attributed to seed noise; variants intended to strengthen the auxiliary did no better. The authors credit the model's competitive scores to its GPT-BERT-style hybrid masked/causal training recipe and EMA weight-averaging, both already established, with training from scratch on the official 10M-word corpus taking 2.3 GPU-hours on a single NVIDIA L40S.", "body_md": "##### Abstract\n\nWe ask whether a low-weight data2vec-style Exponential Moving Average (EMA) self-distillation auxiliary (𝜆=0.1, 𝜏=0.999) improves a small masked language model at the BabyLM-2026 Strict-Small scale. Our base model is a 33M-parameter LTG-BERT encoder with GPT-BERT-style hybrid masked/causal training, trained from scratch on the official 10M-word corpus in 2.3 GPU-hours on a single NVIDIA L40S. A substantial gain in Entity-Tracking during early development from adding the auxiliary motivated the study, but disappeared in later re-evaluation. In a comparison across 6 seeds, no evaluation task changed significantly (aggregate effect p=0.29), and the early gain is best explained by seed noise. Several variants intended to strengthen the auxiliary do no better. We attribute the model’s competitive scores to its hybrid training recipe and to EMA weight-averaging, both already established.\n- Anthology ID:\n- 2026.babylm-main.50\n- Volume:\n- [Proceedings of the Second BabyLM Workshop](https://aclanthology.org/volumes/2026.babylm-main/)\n- Month:\n- October\n- Year:\n- 2026\n- Address:\n- Budapest, Hungary\n- Editors:\n- [Leshem Choshen](https://aclanthology.org/people/leshem-choshen/) ,[Ryan Cotterell](https://aclanthology.org/people/ryan-cotterell/) ,[Mustafa Omer Gul](https://aclanthology.org/people/mustafa-omer-gul/) ,[Jaap Jumelet](https://aclanthology.org/people/jaap-jumelet/unverified/) ,[Tal Linzen](https://aclanthology.org/people/tal-linzen/) ,[Aaron Mueller](https://aclanthology.org/people/aaron-mueller/) ,[Suchir Salhan](https://aclanthology.org/people/suchir-salhan/) ,[Raj Sanjay Shah](https://aclanthology.org/people/raj-sanjay-shah/) ,[Alex Warstadt](https://aclanthology.org/people/alex-warstadt/) ,[Ethan Gotlieb Wilcox](https://aclanthology.org/people/ethan-wilcox/)\n- Venues:\n- [BabyLM](https://aclanthology.org/venues/babylm/) |[WS](https://aclanthology.org/venues/ws/)\n- SIG:\n- Publisher:\n- Association for Computational Linguistics\n- Note:\n- Pages:\n- 631–638\n- Language:\n- URL:\n- [https://aclanthology.org/2026.babylm-main.50/](https://aclanthology.org/2026.babylm-main.50/)\n- DOI:\n- Cite (ACL):\n- Martin Benedikt, Ivan Dostál, and Robin Klubarski. 2026. [Does a Data2vec Auxiliary Help Small Masked Language Models? A BabyLM-2026 Strict-Small Study](https://aclanthology.org/2026.babylm-main.50/) . In*Proceedings of the Second BabyLM Workshop* , pages 631–638, Budapest, Hungary. Association for Computational Linguistics.\n- Cite (Informal):\n- [Does a Data2vec Auxiliary Help Small Masked Language Models? A BabyLM-2026 Strict-Small Study](https://aclanthology.org/2026.babylm-main.50/) (Benedikt et al., BabyLM 2026)\n- PDF:\n- [https://aclanthology.org/2026.babylm-main.50.pdf](https://aclanthology.org/2026.babylm-main.50.pdf)", "url": "https://wpnews.pro/news/does-a-data2vec-auxiliary-help-small-masked-language-models-a-babylm-2026-strict", "canonical_source": "https://aclanthology.org/2026.babylm-main.50/", "published_at": "2026-10-24 00:00:00+00:00", "updated_at": "2026-10-25 21:41:37.677042+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "natural-language-processing"], "entities": ["Martin Benedikt", "Ivan Dostál", "Robin Klubarski", "LTG-BERT", "BabyLM-2026", "Association for Computational Linguistics", "NVIDIA L40S"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/does-a-data2vec-auxiliary-help-small-masked-language-models-a-babylm-2026-strict", "markdown": "https://wpnews.pro/news/does-a-data2vec-auxiliary-help-small-masked-language-models-a-babylm-2026-strict.md", "text": "https://wpnews.pro/news/does-a-data2vec-auxiliary-help-small-masked-language-models-a-babylm-2026-strict.txt", "jsonld": "https://wpnews.pro/news/does-a-data2vec-auxiliary-help-small-masked-language-models-a-babylm-2026-strict.jsonld"}}