Robust Langevin Sampling under State-Dependent Heavy-Tailed Noise with Applications to LLM Fine-Tuning

A new paper proposes three Langevin Monte Carlo methods — robust LMC (R-LMC), adaptive LMC (A-LMC), and robust adaptive LMC (RA-LMC) — for sampling under state-dependent heavy-tailed noise where the noise has fewer than 2 finite moments and satisfies the Blum-Gladyshev (BG-0) condition. The authors establish non-asymptotic convergence guarantees in Fisher information without assuming dissipativity or a functional inequality, and report that RA-LMC matches or outperforms state-of-the-art baselines in accuracy and uncertainty quantification in Bayesian fine-tuning experiments with large language models and vision-language models. R-LMC handles the oracle via geometric-median centering and residual clipping, A-LMC controls discretization error under generalized smoothness with exact gradients, and RA-LMC combines robust estimation with a noise-aware step size.