{"slug": "gem-geometric-entropy-mixing-for-optimal-llm-data-curation", "title": "GEM: Geometric Entropy Mixing for Optimal LLM Data Curation", "summary": "Researchers have introduced GEM (Geometric Entropy Mixing), a framework that reformulates LLM data curation as a variational problem on a hypersphere to overcome flaws in human taxonomies and Euclidean clustering. By decoupling generative priors and optimizing with a Minorize-Maximize algorithm, GEM discovers balanced semantic structures and improves downstream accuracy by up to 1.2% in 1.1B-parameter models. The method establishes a new state-of-the-art when integrated into mixing strategies like DoReMi and RegMix, offering a robust coordinate system for predictable data mixing.", "body_md": "arXiv:2605.26121v1 Announce Type: new\nAbstract: LLM pre-training efficacy increasingly depends on data composition rather than sheer volume. Yet, optimal mixing is hindered by categorization flaws: human taxonomies suffer from ontological misalignment, and Euclidean clustering fails to address embedding anisotropy. We introduce GEM (Geometric Entropy Mixing), a framework reformulating data curation as a variational problem on the hypersphere augmented with a mixing-balance regularizer. By decoupling the generative prior and optimizing the objective via a provable MM (Minorize-Maximize) algorithm, GEM effectively counteracts the cluster collapse to discover balanced semantic structures invisible to Euclidean heuristics. We employ teacher-student distillation to scale this geometric fidelity to web-scale corpora and introduce the Geometric Influence Score (GIS) for interpretable taxonomy generation. Experiments with 1.1B-parameter models demonstrate that GEM establishes a new state-of-the-art when integrated into mixing strategies like DoReMi and RegMix, improving average downstream accuracy by up to 1.2% and offering a robust coordinate system for predictable data mixing.", "url": "https://wpnews.pro/news/gem-geometric-entropy-mixing-for-optimal-llm-data-curation", "canonical_source": "https://arxiv.org/abs/2605.26121", "published_at": "2026-05-27 04:00:00+00:00", "updated_at": "2026-05-27 04:28:15.160780+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "artificial-intelligence", "ai-research"], "entities": ["GEM", "DoReMi", "RegMix", "Geometric Influence Score", "MM algorithm"], "alternates": {"html": "https://wpnews.pro/news/gem-geometric-entropy-mixing-for-optimal-llm-data-curation", "markdown": "https://wpnews.pro/news/gem-geometric-entropy-mixing-for-optimal-llm-data-curation.md", "text": "https://wpnews.pro/news/gem-geometric-entropy-mixing-for-optimal-llm-data-curation.txt", "jsonld": "https://wpnews.pro/news/gem-geometric-entropy-mixing-for-optimal-llm-data-curation.jsonld"}}