{"slug": "what-should-we-remove-evaluating-the-reliability-and-costs-of-rlvr-data", "title": "What Should We Remove? Evaluating the Reliability and Costs of RLVR Data Decontamination", "summary": "An audit of sixteen open RLVR math corpora covering 1,673,514 training items against six benchmarks with 1,949 questions found that at a cosine threshold of 0.80, 52 of 59 sampled removals (88%) were confirmed false removals, while the existing judge-adjudicated decontamination policy achieved 98.1% resolved precision and 98.2% candidate-frame recall on held-out corpora. The study, released as arXiv:2610.21229v1, evaluated eight cleaning policies using 594 independently and blindly dual-reviewed training-benchmark pairs and found no semantic threshold in its preregistered 0.80-0.98 grid met the 90% conservative-precision target on calibration. The authors released reviewed labels, per-policy decisions, and coverage accounting to support evaluating cleaning reliability and cost rather than overlap counts alone.", "body_md": "arXiv:2610.21229v1 Announce Type: new \nAbstract: Reinforcement learning with verifiable rewards (RLVR) math corpora are routinely screened for benchmark overlap, but the reliability and data-loss costs of cleaning policies remain underexamined. We audit sixteen open RLVR math corpora comprising 1,673,514 corpus-specific training items against six benchmarks containing 1,949 questions. We evaluate eight policies based on exact, lexical, and semantic matching using 594 independently and blindly dual-reviewed training-benchmark pairs. Calibration and held-out sets have disjoint benchmark questions and training-example families. A separate review of all 83 candidate matches for 80 sampled training items measures item-level false removal. At cosine threshold 0.80, 52 of 59 sampled removals (88%) were confirmed false removals; this is not a demonstrated population rate. No semantic threshold in our preregistered grid (0.80-0.98) meets the 90% conservative-precision target on calibration. The existing judge-adjudicated policy achieves 98.1% resolved precision and 98.2% candidate-frame recall on held-out corpora. It has zero confirmed false removals among six sampled removals, with one unresolved case; this does not establish zero risk. Within the evaluated candidate frame, this policy offers a more favorable precision-recall balance than the semantic-threshold alternatives tested. Recall does not measure contamination outside the candidate frame. We release reviewed labels, per-policy decisions, and coverage accounting to support evaluation of cleaning reliability and cost rather than overlap counts alone.", "url": "https://wpnews.pro/news/what-should-we-remove-evaluating-the-reliability-and-costs-of-rlvr-data", "canonical_source": "https://www.machinebrief.com/news/what-should-we-remove-evaluating-the-reliability-and-costs-o-x1a3", "published_at": "2026-10-16 04:00:00+00:00", "updated_at": "2026-10-16 05:09:04.604266+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "large-language-models"], "entities": ["arXiv", "RLVR"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-should-we-remove-evaluating-the-reliability-and-costs-of-rlvr-data", "markdown": "https://wpnews.pro/news/what-should-we-remove-evaluating-the-reliability-and-costs-of-rlvr-data.md", "text": "https://wpnews.pro/news/what-should-we-remove-evaluating-the-reliability-and-costs-of-rlvr-data.txt", "jsonld": "https://wpnews.pro/news/what-should-we-remove-evaluating-the-reliability-and-costs-of-rlvr-data.jsonld"}}