{"slug": "a-diversity-diet-for-a-healthier-model-a-case-study-of-french-modernbert", "title": "A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT", "summary": "Researchers at French institutions found that diversity-driven sampling can reduce pre-training dataset size by up to 94% and training time by 73% while maintaining performance in ModernBERT models. In some tasks, diversity sampling improved model quality by up to 10 points over random sampling.", "body_md": "[A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT](https://aclanthology.org/2026.findings-acl.1707.pdf)\n\n[Louis Estève](/people/louis-esteve/),\n[Christophe Servan](/people/christophe-servan-7075/),\n[Thomas Lavergne](/people/thomas-lavergne/unverified/),\n[Agata Savary](/people/agata-savary/)\n\n##### Abstract\n\nDiversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons to investigate theimpact of diversity on pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining atleast comparable performance. We compare diversity-driven sampling algorithms, and we use the best one to pre-train several ModernBERT models on French with a fixed compute budget. We fine-tune and evaluate them on a variety of French benchmarks. We compare them with models pre-trained on randomly sampled data of commensurate size, with the same compute budget. We find that both random and diversity-driven sampling may reduce the pre-training dataset by up to 94% and the pre-training time by up to 73% while maintaining performance. Moreover, in some tasks, the inherent quality of models, estimated via head-only fine-tuning, is up to 10 points higher with diversity sampling than with random sampling.- Anthology ID:\n- 2026.findings-acl.1707\n- Volume:\n[Findings of the Association for Computational Linguistics: ACL 2026](/volumes/2026.findings-acl/)- Month:\n- July\n- Year:\n- 2026\n- Address:\n- San Diego, California, United States\n- Editors:\n[Maria Liakata](/people/maria-liakata/),[Viviane P. Moreira](/people/viviane-p-moreira/unverified/),[Jiajun Zhang](/people/jiajun-zhang/unverified/),[David Jurgens](/people/david-jurgens/)- Venue:\n[Findings](/venues/findings/)- SIG:\n- Publisher:\n- Association for Computational Linguistics\n- Note:\n- Pages:\n- 34168–34181\n- Language:\n- URL:\n[https://aclanthology.org/2026.findings-acl.1707/](https://aclanthology.org/2026.findings-acl.1707/)- DOI:\n- Cite (ACL):\n- Louis Estève, Christophe Servan, Thomas Lavergne, and Agata Savary. 2026.\n[A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT](https://aclanthology.org/2026.findings-acl.1707/). In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 34168–34181, San Diego, California, United States. Association for Computational Linguistics. - Cite (Informal):\n[A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT](https://aclanthology.org/2026.findings-acl.1707/)(Estève et al., Findings 2026)- PDF:\n[https://aclanthology.org/2026.findings-acl.1707.pdf](https://aclanthology.org/2026.findings-acl.1707.pdf)", "url": "https://wpnews.pro/news/a-diversity-diet-for-a-healthier-model-a-case-study-of-french-modernbert", "canonical_source": "https://aclanthology.org/2026.findings-acl.1707/", "published_at": "2026-06-22 00:00:00+00:00", "updated_at": "2026-06-26 08:17:24.553644+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "machine-learning"], "entities": ["Louis Estève", "Christophe Servan", "Thomas Lavergne", "Agata Savary", "ModernBERT", "Association for Computational Linguistics"], "alternates": {"html": "https://wpnews.pro/news/a-diversity-diet-for-a-healthier-model-a-case-study-of-french-modernbert", "markdown": "https://wpnews.pro/news/a-diversity-diet-for-a-healthier-model-a-case-study-of-french-modernbert.md", "text": "https://wpnews.pro/news/a-diversity-diet-for-a-healthier-model-a-case-study-of-french-modernbert.txt", "jsonld": "https://wpnews.pro/news/a-diversity-diet-for-a-healthier-model-a-case-study-of-french-modernbert.jsonld"}}