Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG

Researchers propose TR-RAG, a teacher-regularized reinforcement learning method for cross-lingual retrieval-augmented generation that addresses language drift and unreliable evidence use. The method combines reward optimization with on-policy distillation, improving language consistency and evidence-grounded correctness across three benchmarks. The teacher anchor prevents performance collapses in in-domain languages and improves evidence grounding in out-of-distribution languages.

arXiv:2607.02966v1 Announce Type: new Abstract: Cross-lingual retrieval-augmented generation RAG is often deployed in an English-evidence regime, where users query in diverse languages but retrieved passages remain English. In this setting, generation can fail despite strong base models: English evidence induces language drift English or code-switching outputs and models use evidence unreliably when producing non-English answers. We attribute these failures to two post-training challenges: i errors are prefix-dependent, so fixed-trajectory supervision suffers from prefix mismatch; and ii sequence-level partly discrete / judge-based rewards yield noisy credit assignment and high-variance updates. We propose TR-RAG, a teacher-regularized RL recipe that couples reward optimization with on-policy distillation on student-visited prefixes. A compact student samples on-policy answers, while a stronger frozen teacher is queried only on those prefixes and provides a prefix-wise student-to-teacher reverse-KL anchor. We further introduce a reward decomposition for English-evidence multilingual generation, combining language consistency, character 3-gram recall, and an LLM-judge score for evidence-grounded correctness. Across three benchmarks -- BioASQ-ENKB5, Hotpot-ENKB5, and naturally multilingual MKQA -- and two backbones, TR-RAG improves the composite of language adherence and evidence-grounded correctness over strong baselines. Crucially, the teacher anchor acts as a safety net: on in-domain languages it prevents the large language-consistency collapses up to ~27 percentage points that reward-only RL can suffer by drifting below even the base model, while on distant out-of-distribution languages -- where reward-only RL stalls at the base model's ceiling -- it still improves evidence grounding; and on character 3-gram recall the compact student sometimes surpasses its 70B teacher.