Agentic Data Sampling Framework for Multilingual ASR via Linguistic-Aware Error-Driven Optimization

A new arXiv paper (2610.36538v1) proposes a linguistic-aware agentic data sampling framework that places a data-sampling LLM agent inside the multilingual ASR training loop, driven by a rubric analyzing errors from phonological, morphological and syntactic perspectives. Tested on twelve Common Voice languages with Whisper-medium as the ASR backbone, the method outperformed full-data training while using less training data and lower computational resources, and substantially accelerated training convergence versus conventional sampling strategies.

arXiv:2610.36538v1 Announce Type: new Abstract: Multilingual automatic speech recognition ASR has achieved rapid progress with large-scale pretrained models and multilingual speech corpora. Previous studies have proved the pivotal role of data sampling in multilingual ASR performance. However, existing sampling strategies largely rely on time-consuming manual, heuristic-based designs. To address this, we propose a linguistic-aware agentic data sampling framework that introduces a data sampling LLM agent into the multilingual ASR training loop. The agent is driven by a linguistically grounded rubric that analyzes ASR errors from phonological, morphological, and syntactic perspectives, and uses the resulting feedback together with training statistics, budget constraints, and algorithm profiles to select domain-level, sample-level, and token-level sampling strategies. This design turns evaluation results into adaptive data-selection decisions rather than treating WER/CER as final endpoints only. We conduct experiments on twelve Common Voice languages using Whisper-medium as the ASR backbone. The results show that our method achieves performance better than full-data training while using less training data and lower computational resources, and substantially accelerates training convergence compared to conventional sampling strategies.