KoFinClaim-Abstain: a Korean benchmark that scores LLMs by the cost of false accepts on financial numbers

A new benchmark, KoFinClaim-Abstain, scores large language models on the cost of false accepts rather than raw accuracy, using 500 Korean numeric claims drawn from fictional financial filings where a false accept costs 10 and a human review costs 1. Five open models answering alone passed 7–30% of wrong claims, with EXAONE 3.5 7.8B accepting 36 of 500 and Qwen3 8B accepting 149 of 500, while adding a rule-based checker that re-verifies quotes, numbers, units, periods and conflicts cut Qwen3-14B's false accepts from 46 to 1 in-sample, at the cost of sending many correct answers to review. Answers were hash-sealed before any run and the benchmark is released under CC BY 4.0.

Most LLM evals reward accuracy. In financial documents the expensive mistake is different: confirming a wrong number. I built KoFinClaim-Abstain — 500 Korean numeric claims from fictional filings. For each claim + 2–4 evidence sentences, a system must accept, reject, or hand off to a human. Scoring is by cost: a false accept costs 10, a review costs 1. Five open models answering alone passed 7–30% of wrong claims EXAONE 3.5 7.8B: 36/500, Qwen3 8B: 149/500 . Adding a rule-based checker that re-verifies quotes, numbers, units, periods and conflicts cut Qwen3-14B’s false accepts from 46 to 1 — in-sample, and at the price of sending many correct answers to review. Both caveats are in the card. Answers were hash-sealed before any run. CC BY 4.0. Feedback on the scoring and the labels is welcome.