{"slug": "metadata-free-meta-reweighted-direct-preference-optimization-under-noisy-labels", "title": "Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels", "summary": "Researchers propose a bilevel optimization framework for Direct Preference Optimization (DPO) that recovers clean-data performance under noisy preference labels, using a meta-knowledge-driven method that works without metadata. Experiments on TL;DR summarization and Anthropic HH dialogue show improved alignment over DPO baselines under various noise rates.", "body_md": "arXiv:2607.09796v1 Announce Type: new\nAbstract: Direct Preference Optimization (DPO) has become an important method for aligning large language models (LLMs) with human preferences because it removes the need for explicit reward modeling and reinforcement learning optimization. However, its performance depends heavily on the quality of preference data, and noisy preference data in real-world settings can weaken alignment performance. To address this issue, we propose a bilevel optimization framework and prove, under certain assumptions, that this framework can recover the DPO optimum under clean data. We further derive a prior form for the learnable weighting function under asymmetric label-flipping noise. Considering that high-quality metadata may be difficult to obtain, we propose a task-agnostic meta-knowledge-driven method that enables meta-learning even when metadata is completely unavailable. To reduce the high cost of higher-order gradients in LLM meta-learning, we combine central-difference approximation with LoRA fine-tuning and develop a scalable training scheme. Experiments on TL;DR summarization and Anthropic HH single-turn dialogue show that the proposed method improves training performance over multiple DPO baselines under different noise rates.", "url": "https://wpnews.pro/news/metadata-free-meta-reweighted-direct-preference-optimization-under-noisy-labels", "canonical_source": "https://arxiv.org/abs/2607.09796", "published_at": "2026-07-14 04:00:00+00:00", "updated_at": "2026-07-14 04:23:00.518382+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/metadata-free-meta-reweighted-direct-preference-optimization-under-noisy-labels", "markdown": "https://wpnews.pro/news/metadata-free-meta-reweighted-direct-preference-optimization-under-noisy-labels.md", "text": "https://wpnews.pro/news/metadata-free-meta-reweighted-direct-preference-optimization-under-noisy-labels.txt", "jsonld": "https://wpnews.pro/news/metadata-free-meta-reweighted-direct-preference-optimization-under-noisy-labels.jsonld"}}