RegDivergence-101: An LLM Benchmark for Cross-Jurisdiction Regulatory Contradiction Detection in Life Sciences

Researchers released RegDivergence-101, a 101-pair benchmark for detecting regulatory contradictions between FDA and EMA guidance in life sciences, with an LLM judge (Claude Haiku) achieving 0.830 macro-F1, outperforming lexical (0.511), NLI (0.233), and Graph-RAG (0.663) baselines. The benchmark classifies pairs as AGREE, DIVERGE, or SILENT, with inter-annotator agreement kappa = 0.85, and highlights corpus-level graph construction as the target for large-scale silent detection.

arXiv:2608.28607v1 Announce Type: new Abstract: Pharmaceutical sponsors developing a drug for both the United States and the European Union must reconcile guidance issued independently by the FDA and the EMA. Where the two agencies require substantively the same thing, a sponsor can file once; where they diverge, a single trial design risks rejection in one region; where one agency is silent on a point the other regulates, the sponsor must infer obligations. Today this reconciliation is performed manually by regulatory-affairs experts. We introduce cross-jurisdiction regulatory divergence detection: given an FDA requirement and an EMA requirement on the same topic, classify their relationship as AGREE, DIVERGE, or SILENT. SILENT is inherently directional SILENT FDA vs. SILENT EMA ; we record direction per pair and report per-direction F1 alongside the collapsed label. We release RegDivergence-101, a 101-pair expert-grounded pilot evaluation benchmark labels grounded in three peer-reviewed FDA/EMA comparison studies and primary FDA/EMA/ICH guidance text; dual-annotation inter-annotator kappa = 0.85 , and systematically characterise a four-method baseline hierarchy: lexical heuristic 0.511 macro-F1, 95% CI 0.411-0.605 , NLI cross-encoder 0.233 , obligation-level Graph-RAG 0.663 0.570-0.747 , and flat LLM judge / Claude Haiku 0.830 0.747-0.908 . Three directional observations emerge at pilot scale n = 101 : SILENT is semantically detectable but invisible to entailment-only formulations; pair-level obligation graphs improve over lexical methods but trail flat-LLM context CIs partially overlapping ; and corpus-level graph construction is the indicated architectural target for large-scale silent-detection. RegDivergence-101 is a pilot release establishing the task formulation and baseline hierarchy; four unrepresented regulatory domains and an expansion roadmap are described in Section 7.