{"slug": "robust-langevin-sampling-under-state-dependent-heavy-tailed-noise-with-to-llm", "title": "Robust Langevin Sampling under State-Dependent Heavy-Tailed Noise with Applications to LLM Fine-Tuning", "summary": "A new paper proposes three Langevin Monte Carlo methods — robust LMC (R-LMC), adaptive LMC (A-LMC), and robust adaptive LMC (RA-LMC) — for sampling under state-dependent heavy-tailed noise where the noise has fewer than 2 finite moments and satisfies the Blum-Gladyshev (BG-0) condition. The authors establish non-asymptotic convergence guarantees in Fisher information without assuming dissipativity or a functional inequality, and report that RA-LMC matches or outperforms state-of-the-art baselines in accuracy and uncertainty quantification in Bayesian fine-tuning experiments with large language models and vision-language models. R-LMC handles the oracle via geometric-median centering and residual clipping, A-LMC controls discretization error under generalized smoothness with exact gradients, and RA-LMC combines robust estimation with a noise-aware step size.", "body_md": "<2$ and includes the Blum-Gladyshev (BG-0) condition. We study these relaxations separately and jointly, proposing three methods. Under globally Lipschitz gradients, robust LMC (R-LMC) handles the proposed oracle through geometric-median centering and residual clipping. With exact gradients, adaptive LMC (A-LMC) controls discretization error under generalized smoothness. Robust adaptive LMC (RA-LMC) addresses both relaxations through robust estimation and a noise-aware step size. For all three methods, we establish non-asymptotic convergence guarantees in Fisher information (FI) without assuming dissipativity or a functional inequality. Synthetic experiments demonstrate FI convergence and the benefit of each mechanism. Bayesian fine-tuning experiments with LLMs and vision-language models support our oracle model and show that RA-LMC matches or outperforms state-of-the-art baselines in accuracy and uncertainty quantification.", "url": "https://wpnews.pro/news/robust-langevin-sampling-under-state-dependent-heavy-tailed-noise-with-to-llm", "canonical_source": "https://www.machinebrief.com/news/robust-langevin-sampling-under-state-dependent-heavy-tailed-5f7l", "published_at": "2026-10-12 04:00:00+00:00", "updated_at": "2026-10-12 06:07:53.549284+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "large-language-models", "ai-safety"], "entities": ["Robust Langevin Monte Carlo", "R-LMC", "A-LMC", "RA-LMC", "Blum-Gladyshev condition", "Fisher information"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/robust-langevin-sampling-under-state-dependent-heavy-tailed-noise-with-to-llm", "markdown": "https://wpnews.pro/news/robust-langevin-sampling-under-state-dependent-heavy-tailed-noise-with-to-llm.md", "text": "https://wpnews.pro/news/robust-langevin-sampling-under-state-dependent-heavy-tailed-noise-with-to-llm.txt", "jsonld": "https://wpnews.pro/news/robust-langevin-sampling-under-state-dependent-heavy-tailed-noise-with-to-llm.jsonld"}}