IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law

Researchers introduced IntLawNER, a named entity recognition dataset and benchmark for international law covering 2,987 gold-annotated sentences and 8,094 entity spans drawn from International Court of Justice decisions, UN Security Council resolutions, and European Court of Human Rights judgments. The dataset was built with a hybrid algorithmic-agentic pipeline that reduced 468,000 source sentences to a compact annotation set, with 89.6% of gold spans accepted unchanged from the silver layer, and the benchmark found zero-shot GLiNER collapses to 0.243 micro-F1 on institution-dependent entity types while Claude Opus 4.6 reached the best score of 0.873 micro-F1 with few-shot prompting. The authors report that human-machine agreement metrics can mislead in domain-specific NER, since Cohen's kappa of 0.964 on boundary-matched spans masks a macro-F1 of 0.753 once missing entities, boundary errors, and label corrections are included.

arXiv:2609.22529v1 Announce Type: new Abstract: International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its texts remain without token-level named entity recognition NER resources. We introduce IntLawNER, a NER dataset and benchmark for codified sources of international law, covering 2,987 gold-annotated sentences and 8,094 entity spans from International Court of Justice ICJ decisions, UN Security Council resolutions, and European Court of Human Rights ECtHR judgments, annotated with seven institution-specific entity types. We construct IntLawNER with a cost-effective hybrid algorithmic-agentic pipeline that reduces 468k source sentences to a compact annotation set through candidate retrieval, LLM-based vetting, and human review, with 89.6% of gold spans accepted unchanged from the silver layer. However, the silver-to-gold analysis reveals that human-machine aggregate agreement metrics can be misleading in domain-specific NER: Cohen's kappa=0.964 on boundary-matched spans masks a macro-F1 of 0.753 when missing entities, boundary errors, and label corrections are included. The benchmark shows that zero-shot span-based GLiNER collapses on entity types dependent on institutional function rather than surface form 0.243 micro-F1 , while fine-tuned transformers struggle on rare labels. Carefully selected few-shot examples that demonstrate label contrasts improve every LLM over zero-shot prompting, with Claude Opus 4.6 reaching the best score of 0.873 micro-F1. We release IntLawNER as a benchmark and reusable resource for extracting references in international legal texts.